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ABSTRACT 



One mandate of the Kentucky Education Reform Act of 1990 
requires that students develop a writing portfolio. Ultimately, schools must 
elevate the average performance level of students' portfolios to the 
benchmark of "proficient." During site visits to 29 schools, 36 indicators 
were identified that differentiated writing scores and programs in 
continuously improving and continuously declining schools. The indicators 
were used to develop a self-study needs assessment instrument called the 
"School Study of Writing Instruction." Created as a handbook, the instrument 
enables a school to compare different role groups' perspectives on its 
writing portfolio program and to use the findings to set priorities and plan 
program improvements. This report describes the field test of the instrument 
in 11 schools across Kentucky. Most schools were in rural areas or small 
towns and served a primarily White population; the schools were otherwise 
diverse in terms of size, grade level, and Title I eligibility. Of the 10 
schools completing the study, 9 found it to be a worthwhile experience that 
produced some successful outcomes. Steering committee members and principals 
offered helpful suggestions for improving the handbook. A comparison of 
schools that were or were not assigned facilitators suggests that facilitator 
services were highly beneficial. Overall, the instrument was found to be 
valid and replicable. The potential time investment may temper some 
faculties' motivation to undertake it, but facilitator assistance could 
ameliorate this factor. Appendices include parts of the instrument. 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



evaluation materials, and findings for individual schools. (Contains 22 
references.) (SV) 



O 

ERIC 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



o 

VO 

(N 

Q 

W 



1 

I 

/ 



j 

I 

i 

I 



i 



i 



i 

i 








Field Test of the School Study of Writing Instruction: 
A Self-Study Needs Assessment Instrument 



Wril 



istruction 



Kentucky School 



✓ 



A Collaboration between AEL, Inc. 
and 

Kentucky Department of Education 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 

BEST COPY AVAILABLE 




c 



Field Test of the School Study of Writing Instruction: 
A Self-Study Needs Assessment Instrument 



A Product of the Kentucky State Project: 

Designing Professional Development for Portfolio Improvement 

A Collaboration between AEL, Inc. and the Kentucky Department of Education 



Authors: 

Paige Parrish, Marian Keyes, Sandra Orletsky, Pamelia Coe, Claudia Runge, 
Merrill Meehan, Julia Whitaker, Margaret Nickell, Jean Roberts, Modena Sallee, 
Pamela Ladd, Cathy Caudill, Gaye Foster, Sharon Hatton, Starr Lewis, Shannon Tolbert 



November 1999 



AEL, Inc. 

P.O. Box 1348 
Charleston, WV 25325 



AEL's mission is to link the knowledge from research with the wisdom from practice to improve teaching 
and learning. AEL serves as the Regional Educational Laboratory for Kentucky, Tennessee, Virginia, and 
West Virginia. For these same four states, it operates both a Regional Technology in Education 
Consortium and the Eisenhower Regional Consortium for Mathematics and Science Education. In 
addition, it serves as the Region IV Comprehensive Center and operates the ERIC Clearinghouse on 

Rural Education and Small Schools. 

Information about AEL projects, programs, and services is available by writing or calling AEL. 




Post Office Box 1348 
Charleston, West Virginia 25325-1348 
304-347-0400 
800-624-9120 
304-347-0487 (fax) 
aelinfo@ael.org 
http://www.ael.org 



© 2000 by AEL, Inc. 



This publication is based on work sponsored wholly or in part by the Office of Educational Research and 
Improvement, U.S. Department of Education, under contract number RJ96006001. Its contents do not 
necessarily reflect the views of OERI, the Department, or any other agency of the U.S. government. 



AEL is an Equal Opportunity/Affirmative Action Employer. 



CONTENTS 



Pag', 



EXECUTIVE SUMMARY v 

Background vi 

Findings: How the Field-Test Schools Conducted the School Study vi 

Findings: Need for Facilitator Assistance viii 

Findings: Value, Validity, and Reliability ix 

INTRODUCTION 1 

Objectives 4 

Audience ..4 

METHOD 5 

Data Sources 5 

Participating Schools 6 

Materials 7 

Procedure 14 

SUMMARIES BY FIELD-TEST SCHOOL 16 

Condition 1: Facilitated Schools 16 

Facilitated School #1 16 

Facilitated School #2 16 

Facilitated School #3 17 

Facilitated School #4 17 

Facilitated School #5 18 

Condition 2: Independent Schools 18 

Independent School #1 18 

Independent School #2 19 

Independent School #3 19 

Independent School #4 20 

Independent School #5 20 

Independent School #6 21 

RESULTS 22 

Initiation of the Study Process and Steering Committee Setup 22 

The Interviews (Step 1) 23 

The Report (Step 2) 25 

The Rating Meeting (Step 3) 27 

Priorities and Planning (Step 4) 28 

Assessment of the Steps and Products of the School Study 29 

Time Span and Time Investment 34 



111 




5 



CONTENTS (continued) 

Page 

Comparison of Facilitator/Contact Person Assistance between 

Facilitated and Independent Schools 35 

Assessment of the Worth and Outcomes of the School Study 40 

Summary of the Results 41 

CONCLUSIONS 43 

Value 43 

Facilitator Assistance 45 

Validity and Reliability 47 

RECOMMENDATIONS 49 

REFERENCES 51 

APPENDIXES 

A: Table of Observed Score Differences, t-Values, p-Values, and Significance Levels of All 

Indicators Between the Continuously Improving and Continuously Declining Schools 
B: Indicators of Successful School Writing Programs in Kentucky: 

Executive Summary of Preliminary Findings 

C: Various Data Sources (Principal’s Feedback form, Steering Committee Member 

Feedback form [two versions]. Phone Interview Protocol for Steering Committee Chairs) 
D: Time & Task Summary 

E: Report Form 

F: Rating Guide 

G: Rating Form 

H: School Profile Graph 

I: Findings by Field-Test School 

J: Table of Report Sections and Data 

K: Table of Observed Score Differences, t-Values, p-Values, and Significance Levels of All 

Indicators Between the Continuously Improving and Continuously Declining Schools 
(corrected version to reflect 36 indicators) 

L: Completed Citation Form from 77?e Program Evaluation Standards (1994) 



iv 

6 

o 



LIST OF TABLES 



Page 

1: Characteristics of the 1 1 Field-Test Schools 6 

2; Selected Aspects of School Study Setup by Field-Test School 23 

3; Selected Aspects of Interviewing (Step 1) by Field-Test School 24 

4: Selected Aspects of Report Writing (Step 2) by Field-Test School 26 

5: Selected Aspects of Rating (Step 3) by Field-Test School 27 

6: Selected Aspects of Priority-Setting and Planning (Step 4) by Field-Test School . . 28 

7: Time Estimates per Step of the School Study of Writing Instruction 29 

8: Time Requirement and Helpfulness of School Study Steps by Field-Test School . . 30 

9: Time Span of Field-Test Schools to Complete 

the School Study of Writing Instruction 34 

10: Reported Individual Time Spent to Conduct 

the School Study of Writing Instruction 35 

1 1 : Individuals Investing Substantial Hours in the School Study by Field-Test School . 35 

12: Comparison of Facilitator/Contact Person Services 

Used by Facilitated Schools (FSs) and Independent Schools (ISs) 35 

13: Summary of Lessons Learned from the Field Test of 

the School Study of Writing Instruction 41 

LIST OF FIGURES 

1 : Depiction of Title I Eligibility by Racial Composition per School(s) 7 



V 



7 



EXECUTIVE SUMMARY 



Background 

One of the mandates of the Kentucky Education Reform Act (KERA) of 1 990 is that students 
develop a writing portfolio (Kentucky Department of Education, 1990, 1998b). Ultimately, schools 
in Kentucky are expected to elevate the average rating of students’ portfolios to the benchmark of 
proficient. To aid in this endeavor, the Kentucky Department of Education (KDE) and AEL, Inc. 
(formerly Appalachia Educational Laboratory, Inc.) began a collaborative project in 1996 (AEL, 
1997a). A collaborative research team composed of KDE and AEL staff was formed. The goal of 
this project is to help schools discern the areas for improvement in their writing programs. 

In 1996, the collaborative research team planned a series of site visits to ascertain the 
indicators of quality writing programs. Through site visits to 29 schools, the research team identified 
36 indicators that differentiated writing scores and programs in continuously improving schools from 
those in continuously declining schools. The indicators of successful writing programs address such 
issues as professional development, instructional strategies, administrative support, and family and 
community involvement. 

Based on the indicators that they had identified during the site visits, the collaborative 
research team designed a needs assessment instrument called the School Study of Writing Instruction 
(AEL & KDE, 1999). This instrument, in the form of a handbook, enables a school to compare 
different role groups’ perspectives, about the writing portfolio program. This comparison is made 
available to the entire faculty for school-wide awareness and discussion. Eventually, the results are 
used to set priorities and plan actions for improvement of a school’s writing program. 

A pilot test of the School Study of Writing Instruction was conducted in the fall of 1998 
(Parrish et al., 1999a). It was piloted in its entirety at one Kentucky school, which was offered a 
$ 1,500 stipend by KDE to complete the self-study. The School Study was successful for the pilot- 
test school, ovvdng largely to the faculty’s sense of ownership of the process. Also, the school deftly 
customized the School Study handbook to suit its needs. The outcomes of the pilot test contributed 
formatively to the field test and created the expectation of success in the field-test schools. 

Describing the field test of the School Study of Writing Instruction (AEL & KDE, 1999) is 
the purpose of this report. The objectives of the field test were (1) to describe each field-test 
school’s experience of conducting the School Study, (2) to assess the value of the self-study for 
schools, (3) to assess the level of external facilitator assistance needed to conduct the study 
satisfactorily, and (4) to assess the validity and reliability of the School Study handbook. 

Findings: How the Field-Test Schools Conducted the School Study 

The School Study of Writing Instruction was field-tested in 1 1 schools across Kentucky, and 
each school was offered a $1,500 stipend by KDE. Ten of thel 1 schools completed the School 
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Study to a logical interim point, and 1 school did not complete it. Seven field-test schools conducted 
the School Study within 6 weeks and 3 schools spent 7-9 weeks. According to responses from 43 
Steering Committee feedback forms, each Steering Committee member spent an average of 13 ,5 
hours on the study; about one fourth of these 43 respondents contributed more than 20 hours each. 

There were several telling similarities and differences in how schools orchestrated the School 
Study, which contributed formatively to the collaborative research team’s production of the final 
version of the handbook. 

First, the research team discovered that 3 of the field-test schools included interview data and 
faculty input from a feeder school in their self-studies, a wise strategy for increasing the impact of 
the study beyond their individual schools. 

Most schools were able to complete step 1, the interviews, in 2-3 days. Almost all schools 
solicited more tham one interviewer to complete the task. Yet administrators or faculty members on 
the Steering Committee frequently conducted interviews themselves, rather than delegate the 
responsibility to individuals outside the school. At just 5 of the 10 schools did Steering Committees 
identify only external individuals for interviewing, usually parents or facilitators.' Several schools 
interviewed more teachers and students than suggested, demonstrating their interest in obtaining 
meaningful results from the study. Upon reflection, most Steering Committee members stated that 
the interviews were the most helpful part of the School Study because the comparison between 
teacher amd student interview responses was revealing. 

Several schools’ Steering Committees wrote the report on their writing program, step 2, 
during a meeting rather tham individually. This caused the collaborative research team to modify the 
handbook by including a recommendation to write the report in a meeting setting. Report writing 
was identified by Steering Committee members as the most time-consuming task of the school study 
process, the average time spent per person being 4-5 hours. 

Most field-test schools constructed the rating meeting, step 3, in faculty small groups as 
outlined in the handbook. Three of the 1 0 schools invited faculty to the meeting on a voluntary basis 
£md consequently reported low attendance. Thirty-five of 45 Steering Committee members and 
principals, representing a majority in 8 of the 10 schools, thought that the school profile developed 
at the rating meeting was mostly an accurate picture of their writing program. However, faculties 
at a few schools voiced dissension with the school profile results during discussion. 

Steering Committees at most of the 10 schools assigned step 4 responsibilities, priorities and 
planning, to a committee or combination of committees; Steering Committees at 2 schools assigned 
them to the entire faculty. By the end of the 1998-99 school year, 4 schools had already begun step 
4 with some creative plans in mind, such as hiring writing faculty and involving students in a 



'Moreover, it was not intended that facilitators be asked to conduct interviews . It was hoped that schools would 
find it easier to identify other external agents, such as parents and community members, to perform this task. 
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community project using their writing skills. All 10 field-test schools that carried the study to a 
logical interim point reported the intention to complete step 4 activities in the 1 999-2000 school year. 

Findings: Need for Facilitator Assistance 

The field test was deployed in two conditions of schools — 5 facilitated schools (Condition 
1) and 6 independent schools (Condition 2). Facilitated schools (FSs) had access to a facilitator to 
guide them through the study in person and independent schools (ISs) had access to a contact person 
for telephone consultation. 

Facilitators provided facilitated schools such services as 

• prompting initiation of a school’s study 

• leading an introductory meeting of the Steering Committee 

• selecting interviewee samples 

• training interviewers 

• interviewing 

• facilitating report- writing and rating meetings 

Typically, contact persons provided these services to independent schools only by request. 
Therefore, as intended and expected, facilitated schools received more of these services than 
independent schools. 

Certain aspects of conducting the School Study were compared between FSs and ISs to 
determine whether having a facilitator made a difference in schools’ efficacy to complete it. It was 
found that FSs were more likely to share responsibilities evenly among Steering Committee 
members, to write the school report in a Steering Committee meeting (rather than individually), and 
to have high faculty attendance at the rating meeting. Whereas, ISs were more likely to initiate their 
study without prompting and to proceed to plarming actions (the last step of the study) before the 
close of the 1998-99 school year. In sum, facilitated schools appeared to conduct the study more 
efficaciously than independent schools, although some ISs may have initiated and concluded the 
study more efficiently than FSs. 

When asked to reflect on how much facilitator or contact person assistance they required to 
conduct the School Study, Steering Committee members had an interesting response. Respondents 
from FSs named only those tasks for which they had received help and named none for which they 
had not. Likewise, respondents from ISs named no specific tasks for which they would have 
appreciated help. These findings mean that Steering Committees did not perceive a need for 
assistance beyond what they had already received, demonstrating an exemplary sense of efficacy to 
conduct the study. 

Nevertheless, the collaborative research team maintains that facilitator assistance is 
advantageous and preferable, based on the consideration of all outcomes. First, FSs gave favorable 
appraisals of the help they received from facilitators. Second, despite their overall perception. 
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Steering Committee members at some ISs did request facilitative services from their contact person. Third, 
Steering Committee chairpersons at a few ISs had to act as quasi-facilitators, assuming more responsibility 
than their counterparts at FSs, because they did not have the services of a facilitator (and chose not to 
request such services from their contact person). 

Fourth, the research team identified various errors in field-test schools’ conduction of the study. 
These include failing to gain faculty buy-in before starting, choosing interviewers from within the school, 
interviewing teachers or students in writing, not probing students for fuller responses, not dividing Steering 
Committee duties evenly among members, not enlisting helpers for typing, not requiring faculty to attend 
the rating meeting, and forgetting to do a step in the study. Both FSs and ISs were as likely to experience 
some of these difficulties in implementation, yet ISs were more likely to originate others that may have 
been averted with facilitator help. For the four reasons enumerated, the provision of facilitator services 
has been deemed highly beneficial and desirable for future schools undertaking the School Study. 

Findings: Value, Validity, and Reliability 

The greatest testimony to the value of the School Study of Writing Instruction was the positive 
assessment by the majority of participants. Of the 10 schools that finished the study, participants at 9 
schools found it to be a worthwhile experience that produced some successful outcomes; participants at 
1 school held an unfavorable opinion of the experience. Looking at the results by individual responses: 
of 47 principals and Steering Committee members, 34 said the study was worth the effort and 6 chose to 
reserve their assessment until they witnessed changes in the writing program as a result of the study. Only 
7 respondents said that it was not worth the effort, 5 of these coming from the 1 school with the negative 
experience. 

When asked how the School Study would benefit or affect the school, principals and Steering 
Committee members most frequently responded that writing programs would henceforth become more 
collaborative with revitalized participation by writing nonaccountable teachers. The next most frequent 
response was a general statement of raised awareness among faculty or a general reference to (unspecified) 
changes in writing programs. 

Steering Committee members and principals offered several helpful suggestions for improving the 
School Study handbook, most relating to the interviews step. Mainly, they identified errors that made it 
difficult to transpose interview data to the report and rating form (i.e., a few of the 36 indicators were 
missing from the report and rating forms). Also, about one fifth of respondents recommended that a better 
time of year than late spring be designated for conducting the study. As another quality assurance measure, 
the collaborative research team performed a reliability assessment on the reports that schools wrote. It was 
found that, overall, reports were not exceptionally thorough. Revisions were made to the School Study of 
Writing Instruction handbook to minimize the likelihood of future occurrences of these various concerns. 

Overall, the School Study is evaluated as being valid and replicable. The potential time investment 
required to complete the study could temper some school faculties’ motivation to undertake it. Yet the 
provision of facilitator assistance could help ameliorate this factor as well as benefit schools in other ways. 
The School Study is recommended for ongoing implementation in Kentucky and other schools. 





INTRODUCTION 



One of the mandates of the Kentucky Education Reform Act (KERA) of 1990 is that every 
student develop a writing portfolio for which he/she is accountable at the 4*'', 7*'', and 12*'' grades 
(Kentucky Department of Education, 1994b, 1996, 1999). Coincident with this mandate, the 
Kentucky Department of Education (KDE) developed a writing program. There are four benchmarks 
of achievement by which students’ writing portfolios are judged: novice, apprentice, proficient, and 
distinguished. Schools are required by KERA to achieve an average of proficient on students’ 
writing portfolios by 2010. Since the inception of KERA, teachers have found it possible to move 
students from novice to apprentice, but more formidable to maneuver them to the next benchmark 
of proficient. Presently, many students at varying degrees of mastery have been appraised at the 
apprentice level. (Some teachers have voiced concern that the range of mastery within the apprentice 
benchmark is too wide. They feel that many students deserve to be evaluated at a higher level, and 
that intermediate benchmarks between apprentice and proficient should be created.) 

In 1996, KDE and AEL, Inc. (formerly called Appalachia Educational Laboratory) began a 
collaborative project to assist teachers in writing instruction and, ultimately, to improve their 
students’ writing portfolio scores (AEL, 1995, 1997a). The project, titled the Kentucky State 
Project: Designing Professional Development for Portfolio Improvement, shall be hereafter referred 
to as the Kentucky writing project or project. 

The Kentucky writing project was conceived as an applied research and development effort. 
Prior to the inception of the project, approximately 100 Kentucky schools were identified as having 
shown consistent gains in writing portfolio scores between 1991-1994 (AEL, 1995). Furthermore, 
although some have criticized the writing portfolio scoring process as being subjective, scoring 
audits showed that scores in these schools were more reliable than scores in less successful schools. 
To help less successful schools achieve these same gains, AEL and KDE were interested in a 
preliminary study to identify correlates of score gains in successful schools, possibly followed by the 
creation of new professional development materials and strategies (AEL, 1995). As stated in the 
project proposal, the original objectives of the research and development effort were ( 1 ) to establish 
correlates to gains in writing scores; (2) based on these correlates, to develop and field-test new 
professional development materials to improve both teacher writing instruction and accuracy in 
scoring student portfolios, implement the new materials in a training program, and disseminate the 
materials statewide and nationally; and (3) to determine the applicability of the new professional 
development materials to content areas other than writing instruction. 

In the first meeting between KDE and AEL, the original objectives and activities of the 
project changed substantially (AEL, 1998). Focus converged on the first objective, that of 
identifying correlates of writing score gains, later termed quality indicators or indicators. The 
second and third objectives of designing new professional development were put on hold, because 
key decision makers at KDE agreed that preliminary research into correlates of gains required in- 
depth attention. First, KDE staff proposed that advancing students to the proficient level requires 
a different set of teaching strategies than those sufficient for getting students to the apprentice level 
(AEL, 1998). Second, KDE hypothesized that perhaps variables in addition to professional 
development and teaching strategies were decisive factors in successful schools’ continual 
improvement. Third, KDE believed that the professional development already being provided was 
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of good caliber: it was schools’ level of participation in, not the content of, professional 

development that differentiated schools’ writing programs and performance. For these reasons, KDE 
decision makers were wary of prematurely designing new professional development materials as a 
quick fix for declining schools. 

The initial proposal stipulated that interviews with various school stakeholders would be 
performed in a few schools to develop appropriate interview protocols and to begin the process of 
identifying correlates of gains in writing scores (AEL, 1995). Once the interview protocols were 
refined, a broad telephone survey of successful and unsuccessful schools would be conducted in 
order to establish these correlates of score gains (and, conversely, the correlates of low or declining 
scores). However, the idea of a telephone survey was dismissed at the first KDE- AEL meeting in 
favor of a more penetrating scrutiny. Key KDE decision makers felt that the examination needed 
to be a more pervasive research project of on-site interviews than a telephone survey would afford. 
AEL staff concurred with this recommendation and also encouraged the inclusion of students as one 
of the stakeholder groups to be interviewed. Thus instead of the telephone survey, a series of site 
visits was planned for the purpose of identifying the correlates, or indicators, of quality writing 
programs. 

At the second project meeting, the KDE Writing Program staff, the KDE regional writing 
consultants (then supervised by KDE, now by regional service centers), and various AEL staff 
formed a collaborative research team, also referred to as the research team or team. To identify the 
indicators of quality writing programs, the team decided to first make site visits to schools that were 
continuously improving in their writing program. 

In preparation for site visits, the collaborative research team identified probable indicators 
as suggested by elements of Kentucky’s writing program, wrote interview protocols based on them, 
tested the protocols during exploratory visits to 7 schools not in the intended sample, and then 
refined the protocols. For choosing the sample, the team created the designations continuously 
improving and continuously declining to define schools that met a certain criterion: continuous 
improvement (or conversely, continuous decline) in KDE writing scores over a 4-year period.^ Next, 
the site visits for the purpose of identifying the indicators were made to 22 of 43 identified 
continuously improving schools.^ Based on these activities, 34 indicators of successful writing 
programs were identified.'* 

Next, site visits were made to 7 of 18 identified continuously declining schools to discover 
how their writing programs fared relative to the indicators in comparison to the improving schools 



a 3-year period (the typical period of assessment in Kentucky) had been used, the pool of continuously 
improving schools would have numbered 100; however, the use of a 4-year period narrowed the pool to 43 schools 
(AEL, 1997a). 

^Two of the schools in the sample later merged, thus changing the sample size to 21 of 42 schools. 

^Actually, 36 indicators were identified; however, two were inadvertently omitted from the field-test version 
of the handbook. Since the field-test schools were equipped to investigate only 34, this number will be mostly used. 
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(these 7 being different than the 7 schools visited during the initial exploratory visits). The 
differences in the scores between the continuously improving and declining schools were 
subsequently tested for significance; the score differences were found to be statistically significant 
on 33 of the 34 indicators^ (see Appendixes A and B; AEL, 1997b; Coe et al., 1999a). 

In sum, through site visits to a total of 36 schools, more than 100 teachers, 200 students, and 
50 administrators were interviewed to help establish and verify the 34 indicators (AEL, 1997b). 
These indicators encompassed issues such as administrative support, writing program coordination, 
and family and community involvement, in addition to the expected salient issues such as 
instructional strategies and professional development.® 

The strong emergence of a group of indicators related to teacher participation in professional 
development confirmed the wisdom of having earlier suspended one of the original objectives of 
creating new professional development programs. In testing these indicators, it was found that 
schools in which teachers had participated in professional development opportunities already 
available through KDE were more likely to evidence improved student writing than schools with 
untrained teachers. Thus the more salient need related to these indicators appeared to be increasing 
teacher participation in existing professional development, not revamping the programs themselves. 

The second activity to emerge out of the modified objectives of this project was the 
development of a comprehensive needs assessment instrument. The instrument originated from the 
collaborative research team’s earlier activities related to identifying the indicators; it was iterative 
in conception in that it grew out of using the interview protocols and writing the reports of findings 
from each school. While doing these activities, the research team (1) confirmed that talking to 
students was indispensable in evaluating a school’s writing program and (2) decided that performing 
the tasks of the School Study would be a more valuable awareness experience for schools were they 
to do it themselves. The team envisioned an instrument whereby schools, led by an in-house 
faculty-based Steering Committee, could assess for themselves the strengths and weaknesses of their 
writing program relative to the indicators, with key operational elements being input from all school 
stakeholders and self- (versus external) evaluation. Thus was conceived and developed a self-study 
needs assessment for schools called the School Study of Writing Instruction (AEL, 1997a; AEL & 
KDE, 1999). Henceforth, it shall be referred to as the School Study, self-study, study, handbook 
manual, or instrument. In accordance with the intent of KDE and AEL decision makers, the 
collaborative research team had designed an insightful approach for eliciting the voices of all school 



^The only nonsignificant indicator was Evidence teachers write with students. Its nonsignificance was 
attributed to two skewed means among the group of continuously declining schools. (As explained in Footnote 4, the 
correct number of indicators is 36: thus, to speak accurately, score differences on 35 of 36 indicators, not 33 of 34, 
were significant.) 

®Only low, nonsignificant correlations were found between writing portfolio scores and characteristics such 
as class size, school size, number of grades, and socioeconomic status (SES) (AEL, 1998; Coe et al., 1999a). In other 
words, writing improvement does not bear a noticeable relationship to school stmcture or demographic makeup of 
schools. Thus, it is the actions that schools do or do not take relative to the indicators, rather than any unchangeable 
characteristics that schools possess, that formatively contribute to their improvement or decline. 
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stakeholders, particularly those of students. This School Study is the instrument under consideration 
in this report. 

In this report, attention is given to the prospect of providing external assistance to schools 
that may conduct the School Study in the future. Based on observations made during site visits, 
technical assistance emerged as a potential need for schools to successfully conduct the study. 
Because facilitator assistance was deemed beneficial in the pilot test, two conditions of such 
assistance were employed in the field test to further gauge this need: 5 schools were assigned a 
facilitator from the collaborative research team who would be regularly present and involved with 
a school as it performed the tasks of the study (facilitated schools) and 6 schools were assigned a 
contact person who would be available for telephone consultation (independent schools). 

The remainder of this report describes the field testing of the School Study of Writing 
Instruction in 1 1 schools across the eight regions of Kentucky, selected to represent different grade 
levels, enrollment sizes, and geographic and demographic regions. Each school was offered a $ 1 ,500 
stipend to be granted upon completion of its self-study. The field test concluded in the summer of 
1 999 and implications for the instrument’s final implementation are considered. This instrument has 
the potential to be widely used to build schools’ capacity for improvement in writing instruction. 

Objectives 

The four objectives of this report are to 

1 . describe each field-test school’s experience of conducting the School Study of Writing 
Instruction 

2. assess the value of the School Study for schools desiring to improve their writing 
program 

3. assess the level of facilitator assistance required by schools to conduct the self-study 
satisfactorily 

4. assess the validity and reliability of the field-test version of the School Study handbook 
for the purpose of improving the final version (AEL, 1997a) 

Audience 

The primary audience for this report is the collaborative research team of the Kentucky 
writing project: the Kentucky Department of Education Writing Program staff, the Kentucky 
regional writing consultants, AEL staff members assigned to the project, and AEL consultants. A 
secondary audience is the Kentucky State Caucus of the AEL Board of Directors, which acts as an 
advisory group to this project. Other audiences include those interested in needs assessment 
instruments, specifically school self-studies of curricular programs, and those interested in school 
writing programs. 
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METHOD 

The method is divided into two conditions. In the facilitated condition, 5 schools were 
assigned a facilitator from the collaborative research team to lead them in the School Study. In the 
independent condition, 6 schools were assigned a contact person from the team to be available for 
telephone consultation. The data sources, participants, materials, and procedure are described next. 

Data Sources 

The data sources used to complete this report are described below. 

Steering Committee Member Feedback form. A form of 1 1 questions accompanied the 
mailing of the field-test materials. The research team requested that it be distributed to each 
participant on the Steering Committee upon completion of the study. Questions related to 
time estimates for completing the study, the nature of planning activities, perceived need for 
facilitator assistance, and the utility of the study (see Appendix C). The facilitator assistance 
question was customized to each of the two conditions of schools. Forms were returned to 
the research team between late May and late June 1999. 

Principal’s Feedback form. A brief form of 4 questions also accompanied the original 
materials with the request that the principal complete it at the conclusion of the study. All 
questions centered on the utility of the study (see Appendix C). Forms were returned to the 
research team between late May and July 1 999. 

Phone interviews with Steering Committee chairs. The Steering Committee chair at each 
field- test school was contacted by a member of the collaborative research team and asked 
several questions regarding the performance of the specific steps of the School Study (see 
Appendix C for the interview protocol). Because a few schools were not finished with the 
study by the date of the first phone call, some follow-up calls were necessary. The phone 
interviews were conducted between mid-May and mid- July 1999. 

Facilitator logs. Facilitators each kept a log of interactions with their assigned school(s) to 
track their progress in carrying out the School Study. Some contact persons (for the schools 
in Condition 2) kept a log as well, although they were not asked to do so. These were 
returned between June and July 1999. 

Roundtable debriefing meetings of the collaborative research team. At two meetings in 
May and July 1999, near the completion of the field test and at its conclusion, facilitators, 
contact persons, and other members of the team shared their insights about schools’ progress. 

Phone and e-mail contacts with facilitators and contact persons. Members of the 
collaborative research team who did not serve as a facilitator or contact person 
communicated with those who did to check schools’ progress and gather data. 
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Participating Schools 

A pool of 1 2 schools was developed by nominations of the collaborative research team. The 
field-test nominations were purposefully selected from the eight education regions of Kentucky to 
represent different enrollment sizes, grade levels, and geographic and demographic regions (see 
Table 1 ). The research team was interested in discovering the School Study ’s utility in various school 
contexts because writing portfolio requirements apply to all schools. The Kentucky Commissioner 
of Education sent an invitation letter to the nominated schools, which included an offer of a $ 1 ,500 
stipend, to which 1 1 schools responded. 

Diverse representation is evident for most characteristics shown in Table 1 except locale type 
and race. (Location is more diverse than it appears given the geographic and education region 
demarcations of Kentucky.) Regarding locale type, most field-test schools came from small towns.’ 
None came from the two largest locale types — a large city or an urban fi^inge of a large 
city — however, one was in a mid-size city. Regarding race, most schools came fi-om a district with 
a majority white population. Although this sample is not extremely diverse in terms of locale type 
and race, it is believed to be to an extent that is representative of the state. 



Table 1: Characteristics of the 1 1 Field-Test Schools 



Enrollment 


Grade Levels 


Location/KY 


Locale Type® 


Race’’ 


Title I” 


# schools 


# schools 


# schools 


# schools 


# schools 


# schools 


0-300 


PreK-6 


North or NW 


Rural area 


> 95% white 


1 - 10% 


3 


3 


2 


2 


8 


2 


301 -600 


PreK-8 


East or SE 


Small town 


88% W; 10%B 


1 1 - 20% 


4 


3 


6 


6 


1 


3 


601 - 900+ 


Middle 
(4-8; 7-8 gr.) 
2 


Central 

1 


Urban fringe of 
mid-size city 
2 


80% W; 18%B 

1 


21-30% 

2 


4 


High 

(7-12; 9-12 gr.) 
3 


Western 

2 


Mid-size city 

1 


60% W; 39%B 

1 


31-50% 

4 



Note. Read each column of data as a separate entity; there is no relationship between successive cells in a row. 
"Data are from Burczyk (1998). 

'’Data are from Quality Education Data (1998). Percentages apply to the district in which each school resides. 



’These are Johnson’s locale type definitions in abbreviated form (Burczyck, 1998): 

A rural area has a population of fewer than 2,500 inhabitants. 

A small town has a population of 2,500 to 24,999. 

A large town has a population of at least 25,000. 

An urban fringe of a midsize city is a place within the metropolitan area of a mid-size city. 

A midsize c/ty has fewer than 400,000 inhabitants and a population density smaller than 6,000 people per square mile. 
An urban fringe of a large city lies within the metropolitan area of a large city. 

A large city has at least 400,000 inhabitants and a population density of at least 6,000 people per square mile. 
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Figure 1 shows the latter two characteristics of Table 1 — race and Title I eligibility — in 
relation to each other. 

Figure 1; Depiction of Title I Eligibility by Racial Composition per School(s) 



8 schools 
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1 school 


1 school 


1 school 


Title 1: 




Title I: 




1 

Title I: ' 


11% 




21% 




31% ! 



>95% white 



88% white & 
10% black 



80% white & 
18% black 



60% white & 
39% black 



Racial Composition 



Note. Locale type data are from Burczyk (1998) and other data are from Quality Education Data (QED) (1998). 
Percentages reflecting race and Title I eligibility apply to the district in which each school resides. 

Materials 



Since the School Study of Writing Instruction (AEL & KDE, 1999) is new, care is taken in 
this section to describe it accurately. The study is a process divided into four main steps: (1) the 
interviews, (2) the report, (3) the ratings, and (4) the priorities. The interview questions, report 
topics, and rating categories correspond to the 34 quality indicators of improving writing portfolio 
scores. They deal with issues of administrative support, professional development, school climate, 
writing program coordination, focus on writing portfolios and writing, in general, instructional 
strategies, family and community involvement, and assessment of the writing program’s value for 
students. The interviews are the data upon which all subsequent analyses are based. They are the 
linchpin of the study; because they come from four different role groups (district administrator, 
principal, teachers, and students), they are a medium for closely comparing different parties’ 
experiences and perceptions of the writing program. 



8 



Content validity of the School Study instrument was determined in two stages, each 
consisting of several Steps. The first stage involved identifying the'variables for examining writing 
programs (Coe et al., 1999a, 1999b). First, indicators were initially derived frofn the piarameters of 
Kentucky’s writing program (see KDE, 1994a). Second, the collaborative research team developed 
draft interview protocols around the initial indicators, tested them in exploratory visits to 7 Kentucky 
schools, and refined them. Third, the research team identified 43 schools with consistently 
improving writing scores and 1 8 with consistently declining scores. Based on site visits to 22 of the 
improving schools and 7 of the declining schools, in which more than 100 teachers, 200 randomly 
selected students, and 50 administrators were interviewed, team members established 34 indicators 
of successful writing programs (see Footnote 4 and Appendix A) (Coe et al., 1999a, 1999b). A 
comparison of the difference in mean scores between the 22 schools with improving scores and 7 
schools with declining scores verified the indicators’ capability to differentiate writing programs. 
These steps encapsulate the process by which the indicators were validated. 

The second stage of establishing validity involved creating the School Study of Writing 
Instruction as a usable instrument for schools. First, it was designed around the indicators identified 
in the research stage. Second, it was designed in such a way to foster the involvement of several 
school stakeholder groups in studying a school’s writing program, particularly the inclusion of 
students as informants of writing instruction. Finally, to enhance utility, the School Study of Writing 
Instruction was, reyised through several iterations based on meetings of the collaborative research 
team, on the pilot test, and on this field test. 

The School Study handbook is a process manual in a three-ring binder; It includes an 
introductory overview and letter to the Steering Committee, as well as a section for each of the four 
main steps listed above. The forms for each step are compiled on computer disk. The overview 
includes a “Time and Task Sumrnary” that provides estimated completion times for each of the 
activities in the four steps (see Appendix D). Confidentiality and the intention of in-house use of 
the information gathered are emphasized several tirnes throughout the manual. The entire process 
as outlined in the manual is suinmarized below. ' 

Engaging faculty buy-in and forming the steering committee. Before the study is begun, 
it needs to be introduced and supported by the faculty. If the faculty express a wish to undertake the 
study, a Steering Corhinittee is foimed. The Steering Committee is responsible for planning — and 
carrying out some' of— the steps in the study, which culminates in a faculty-wide school analysis 
conducted during one or two meetings. The manual estimates “one month with focused attention” 
as the time frame for completion. The Committee is composed of at least four individuals. To 
ensure diverse representation, it should include' at least one teacher who either leads the writing 
program or is portfolio-accountable, one whb is portfolio-nonaccountable by virtue of the grade level 
that the person teaches (i.e., not 4“', 7“',' or n* grades), and one who is nonaccountable by virtue of 
the content area that the person teaches (i.e., a subject other than language arts). A parent may be 
included, preferably a member of the School Based Decision Making (SBDM) Council if one exists. 
If feasible, a community member possessing some familiarity with the writing program may be 
invited. After formation, the Steering Committee;chooses;a Chair. • . 
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Planning meetings and other preparation. Although implied that the Steering Committee 
would have to meet to plan the course of the study, no specific guidance is provided in the handbook 
about how best to accomplish this, except for periodic reminders to have sufficient numbers of 
relevant forms copied prior to the various steps in the process. 

The four steps of the self-study process. A school’s exploration of its writing program is 
a progression from interviews to a report to a rating session (and school profile) to setting priorities 
and planning actions. The purposes of the process are to highlight and verify the school’s most 
distinct needs and to galvanize the entire faculty’s interest in improving the writing program. 

1 . The Interviews . The interviews represent the data collection phase of the self-study. 
There are four role groups to be interviewed: students, teachers, the principal, and a district-level 
administrator. Depending on the size of the school, 10-14 students and 3-10 teachers are to be 
interviewed. These numbers for interviewee samples were designated based on their practicality (in 
terms of both time and data management), as established by the collaborative research team during 
the site visit phase of the project. 

The Steering Committee is encouraged to select interviewees in the teacher and student role 
groups by stratified random sampling according to grade level(s) and content area(s). For the teacher 
sample, the school’s writing leader(s) are automatically included. Then other teachers’ names are 
drawn at random from various bowls. To ensure that a diverse sample is interviewed, teachers’ 
names are distributed among different bowls before random drawing occurs, according to whether 
or not they are portfolio-accountable. A variety of content areas, grade levels, and tracks should be 
represented in the final sample. 

Then the students are selected randomly, but only from the classes of the teachers who were 
themselves selected as interviewees. The student interviewees need to come from the classes of the 
teacher interviewees for meaningful comparison and contrast of responses about writing program 
practices. A range of numbers representing the largest class is placed in a bowl. As a number is 
drawn from the bowl, the name corresponding to that number on the first teacher’s roster is the 
student chosen to be interviewed. The next number drawn is matched to a student’s name on the 
next teacher’s roster, and so on. If a number drawn is higher than the number of students in a 
particular class, another number is drawn. Alternates are drawn in the event that a student 
interviewee is absent on the day of the interviews. 

The principal interviewee is the/a school principal. Finally, the district administrator 
designated as the interviewee should be the administrator most closely involved with the writing 
program. 

After selecting the interviewees, the Steering Committee recruits other individuals to serve 
as the interviewer(s) of the role groups. The interviewer(s) should not be well-known to the 
interviewees. Possible interviewers are teachers from other schools, district or regional staff, 
experienced interviewers from the community, or college students. Even middle or high school 
students from other schools may be selected to interview students below their grade level. 
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If the Steering Committee desires, two Committee members (versus the interviewer [s]) may 
conduct the principal and/or district administrator interviews. The primary benefit in doing so would 
be that the Steering Committee members could gain valuable insight into administrator perspectives; 
a potential disadvantage would be bias on the part of the interviewer. 

The Steering Committee should train the interviewer(s) in proper techniques and 
confidentiality according to their level of interviewing expertise, helping with practice sessions or 
a few joint interviews as necessary. If possible, it is recommended that interviewers work in pairs, 
so that one person can ask the questions while the other records responses and operates a tape 
recorder. One of the most important instructions to interviewers is to read over the interview forms 
prior to conducting any interviews. Another major instruction is to avoid being repetitive — in 
particular, to avoid asking a question if the interviewee has already answered it via an unanticipated 
response to a previous question. Based on the experience of the research team during site visits, it 
is recommended that students be interviewed in groups of two or three and teachers individually. 
It was observed during these visits that students spoke more freely in the company of their peers than 
in individual interviews. Also, group interviews saved time. On the other hand, it was believed that 
teachers responded more freely in their interviews to questions about their practices and those of the 
faculty in general, because these interviews were conducted confidentially and individually rather 
than in groups. 

In the pilot test version of the School Study handbook, the interviewer was given the option 
of providing teachers with the interview form and asking them to complete it in writing as if it were 
a survey in lieu of interviewing teachers orally. However, the pilot school Steering Committee 
decided against this option, speculating that teachers would deliberate too long and write responses 
too verbose for Steering Committee members to incorporate into the school report. The 
collaborative research team readily agreed with this assessment, since the team also had concerns 
about the ability to elicit spontaneous and valid responses in a timely manner were interviews given 
as surveys. 

Thus in the field-test version of the handbook, the survey was designated as a last resort 
option only. An example of a last resort option would be providing a survey to a teacher only after 
several failed attempts to schedule a verbal interview due to time pressures and the realization that 
the absence of her/his interview was stalling the progression of the study. This option was not 
exercised during the site visit research phase or during the pilot test, for the reasons just discussed; 
however, it was retained as an option in the field-test handbook, with emphasis on last resort, as a 
means of preventing schools from adopting this method as a viable, efficient alternative to 
interviewing. 

The interview form for teachers has 47 questions; for students and the principal, each 32 
questions; and for the district administrator, 16 questions. The estimated time to conduct an 
interview is 30-45 minutes; one interview per 50-minute class period allows time for regrouping 
between interviews. The estimated total time spent by the interviewer(s) is 10-14 hours, depending 
on the number of interviewees. 
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The purpose of the interviews is to gain a comprehensive body of information for use as raw 
data. People from the four role groups answer questions pertaining to similar topics. I’he level of 
agreement within and among groups yields cardinal information about the sophistication and 
coordination of the writing program, which is then translated to the report in step 2. 

2. The Report . The report, written by the Steering Committee, is a synthesis of responses 
from the 15-26 interviews (see Appendix E). On the report form, the 34 indicators are recast as 
report topics and grouped under 1 7 headings. Under each topic on the report form, space is provided 
for the response of each role group relevant to that topic, (i.e., for the report topic, “Family 
Involvement,” what did the district administrator have to say? the principal? the teachers? the 
students?). As appropriate, some report topics call for the response of only some role groups, such 
as “Professional Development,” which calls for the response of all groups except students. Also, a 
space labeled “Other” is provided under each topic for report writers to add salient firsthand 
Imowledge that did not surface in the interviews. Because this exercise involves merging the 
information given by various individuals in the separate interviews, the interview data begin to form 
an integrated picture of the writing program at this step. 

A large number of interview responses have to be incorporated meaningfully into a report 
(there are 1 6-47 questions per each of 1 5-26 interviews); therefore, three aids were provided to field- 
test schools for writing the report. 

The first aid was the removal of a data reference table called the “Table of report sections and 
data.” The table had been included in the earlier pilot version handbook, designed to help link 
interview questions to report topics. However, the pilot test school had experienced difficulty using 
the table because question prompts were not listed in enough detail. Yet the research team had 
difficulty detailing them any further because (1) many questions applied to more than one topic and 
(2) whether they were relevant or not often depended on interviewees’ responses. For this reason, 
the research team decided to test the possibility that Steering Committees might fare better without 
a table of question prompts. Therefore, the table was removed for the field test.® 

The second aid consisted of two sample reports intended to give report writers an idea of the 
report’s scope. One sample was written by the research team; the other sample was the pilot test 
school’s actual report, the. pilot school having agreed that it could be included as a sample in the 
field-test handbook. 

The third aid was a suggestion to compile the teacher interview responses and student 
interview responses. In the pilot test, the Steering Committee had devised an extremely powerful 
method for consolidating the interview data so that it could be manageably read and assimilated into 
the report: the Steering Committee chair typed all teacher interviews onto one teacher interview 
form, labeling responses, “Teacher 1, Teacher 2 . . .,” and so on, and all student interviews onto one 



*The table actually remained in the field-test handbook in another form for another purpose — that of dividing 
and assigning report sections among report writers. But the question prompts had been removed, so the table no longer 
served the purpose of linking topics to interview questions. 
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student interview form, distinguishing their responses in a like manner. In sum, 6 teacher interviews 
and 10 student interviews were compiled onto one form each (while the 1 principal interview and 
1 district administrator interview were each left as is); this meant that only 4 interview forms, instead 
of 1 8, had to be consulted by report writers (i.e., the aggregated teacher interview, aggregated student 
interview, principal interview, district administrator interview). The collaborative research team 
valued this as a helpful outcome of the pilot test and so incorporated guidelines into the field-test 
version handbook for compiling interviews. 

To help the Steering Committee divide report writing among themselves, the 34 report topics 
are divided among 8 sections (see Footnote 8). It is recommended that no Steering Committee 
member be assigned more than 2 sections to prevent any one individual fi'om receiving a burdensome 
share of the task. If necessary, the Committee can enlist other faculty to join in writing the report. 
The handbook instructs the report writers to first scan each interview (or compiled interview) in its 
entirety and then answer each report topic. It suggests that each report topic response should 
summarize the common view and note any divergent views. 

The guidelines predict report writing to be the step that takes the most time for the Steering 
Committee. The estimated time is 1-2 hours for two report sections; thus the estimated cumulative 
time spent by all writers is 4-8 hours. The manual suggests that when Committee members finish 
their assigned sections, they consider hiring someone or soliciting a volunteer to type the complete 
report. 



The purpose of the report is to create a portrait of the school’s writing program that highlights 
strengths, weaknesses, consistencies, and inconsistencies. In the indicator identification phase of 
this project, the team found that high consistency among role groups’ responses to interview 
questions was prevalent in improving schools (AEL, 1997b). Conversely, the team expects 
incongruous responses within and among role groups to reveal the tenuous elements of a school’s 
writing program. Student interviews especially are integral to uncovering problematic elements: 
their customer experience of the writing program compared to teachers’ descriptions likely reveals 
more than just descriptions given by teachers and administrators. 

3. The Ratings . The full faculty is convened by the Steering Committee to produce the 
school’s ratings, which are based on the report. A few days prior to the meeting, the Steering 
Committee (or clerical helpers) will copy and distribute the completed report to all faculty, 
requesting that they read it before the meeting. 

At the meeting, faculty are divided into groups of three to five participants. Smaller schools 
may have only two individuals per group; the number per group should not exceed five, even in 
larger schools, to better insure the likelihood that everyone will contribute. The small group size of 
three to five individuals was determined by the research team based on their collective experiences 
as educators and facilitators. The groups should be interdisciplinary, mixed by department and grade 
level, so that substantive cross-discipline discussions of writing can occur. To the degree possible. 
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each group should include a language arts or writing program knowledgeable person. Each small 
group then chooses a group leader and a recorder. 

Four documents are used to mark ratings; 

a. the report, which each individual has brought with her/him (Appendix E) 

b. a rating guide, excerpts of which are given to each individual according to the sections 
assigned to each group (Appendix F) 

c. a blank rating form, given to each individual (Appendix G) 

d. a blank school profile (Appendix H) 

The rating guide describes schools operating at high, medium, and low performance for each of the 
report topics on a 3-point Likert-type scale derived from site visits, and includes authentic field- 
based examples. The rating form consists of a 7-point Likert-type scale per topic ranging between . 
low and high. Finally, the school profile is a bar graph representation of the rating form results. 

Prior to faculty breaking into small groups, the Steering Committee briefly entertains 
questions about the report. Then the method for rating the topics is explained, including how to 
resolve discrepant ratings and reach consensus. In the manual, this exercise in discussing differences 
is strongly emphasized as a means in its own right for facilitating better understanding of the writing 
program among staff. 

For rating purposes, the report is divided into sections equal to the number of small groups. 
Once in groups, participants read the examples in the rating guide, compare them to the 
corresponding topics in the report, discuss them, give a rating to each topic in their assigned section, 
and record the results on their rating forms. 

If time allows, the meeting leader can read each group’s ratings aloud so that everyone can 
complete her/his form and discuss the ratings as a whole group. Each group should be prepared to 
discuss its findings, particularly the topics on which group members had difficulty reaching 
consensus. If there is no time left, the Steering Committee will develop the school profile based on 
the composite ratings and distribute a copy to everyone. If the Steering Committee has decided that 
the full faculty (versus a committee) will set the writing program priorities, the next meeting should 
be announced. Based on the experience of the pilot test school, the estimated time for the rating 
session is 2-3 hours. 

The explicit purpose of the faculty rating session is to apply a quantitative interpretation to 
the qualitative analysis given in the report so that it can be more easily condensed into a 
parsimonious visual representation of the data. The implicit, yet more important, purpose is to fully 
engage faculty in discussion of their writing program. 
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4. The Priorities . This step is comprised of two activities: setting priorities and planning 
actions. The Steering Committee may choose one of two groups to set the school’s priorities — either 
a designated committee, such as the School Consolidated Planning Committee, or the full faculty. 
Whichever option is chosen, requisite materials include the completed school profile, the report, the 
rating guide, and guidance questions for targeting indicators as improvement priorities. If the 
committee option is chosen, committee members receive additional guides such as examples of 
effective practices from other Kentucky schools and the Resource Inventory completed by the 
principal. 

If the faculty option is chosen for setting priorities, the Steering Committee schedules 
another faculty meeting. At the meeting, faculty break into cross-content, cross-grade groups of five 
to further reinforce the practice of faculty from all disciplines and grades taking responsibility for 
the writing program. They are instructed to target one to three indicators for improvement. Each 
group’s choices are reported to the whole body and displayed on large paper. Next, weighted voting 
takes place, with each participant placing between one and three adhesive dots beside their top 
priorities. For instance, each participant receives three dots; they could place all three of their dots 
by one indicator or place one dot each by three different indicators. The three indicators with the 
most dots are selected as priorities. 

Whether a committee or the faculty sets priorities, participants should base their judgments 
on the data and guidance materials. The manual gives suggestions about how to make informed 
decisions in targeting priorities. The main suggestion is to target indicators that are most meaningful 
to the school, rather than the lowest-rated indicators just because they are the lowest. Two 
completed school profiles, one an average of the continuously improving schools studied and the 
Other an average of the continuously declining schools, are supplied in the manual for reference; they 
are not intended for use as standards by which to target priorities for a specific school. 

For the second part of step 4, planning actions, the handbook recommends that a designated 
committee such as the School Consolidated Planning Committee fulfill this task. The committee 
plans actions based on the priorities set by them or the faculty, using any relevant materials 
mentioned previously. 

Setting priorities and planning actions are the culmination of the School Study. Their purpose 
is to give direction to improving the writing program. 

Procedure 

Two copies of the School Study of Writing Instruction handbook were mailed to each of the 
1 1 schools that had agreed to participate, along with a version on disk in WordPerfect 8.0, in late 
January 1999. The materials were accompanied by an introductory letter from the research team to 
the principal. As an incentive to field test the instrument, a stipend of $1 ,500 was provided by KDE 
to each school. Also, a letter detailing procedures was mailed to facilitators and contact persons. 
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Six KDE writing consultants (members of the collaborative research team) were assigned 
among the 1 1 field-test schools to serve as either facilitator or contact person, each assigned to one 
or two schools in her region. Five schools had a facilitator, who would be regularly present and 
involved with guiding the study (Condition 1, facilitated schools), and 6 schools had a contact 
person, who would be available for telephone consultation (Condition 2, independent schools). 
Facilitators and contact persons began calling their assigned schools in late January 1999 to set up 
meetings or to answer schools’ questions and check their progress regularly. During and at the 
conclusion of the field test, feedback was gathered from field-test participants via the several data 
sources described earlier. 

Each field-test school’s experience of conducting the School Study was assessed on whether 
or not it produced some successful results. In this report, successful results means that (1) a school 
completed the study to a logical interim point with adequate attention given to at least the first three 
steps, (2) a majority of principals and Steering Committee members found the study to be a 
worthwhile experience, and (3) principals and Steering Committee members anticipated benefits or 
changes to the writing program as a result of doing the study. The quality of the methods used by 
a school to carry out its study was not Judged relative to successful results; however, the suitability 
and quality of schools’ methods are discussed. Also, some schools experienced factors such as 
attrition by Steering Committee members and less than total faculty participation at the rating 
meeting; however, these factors were not counted against the success of these schools because doing 
so would negate the accomplishments of the individuals who did participate. 

In addition, some schools gave permission for their findings to be reviewed by the research 
team so that a reliability assessment of the School Study handbook could be made. Specifically, the 
interviews and report were requested from each school for use in assessing how efficaciously field- 
test schools transcribed interview data to the report. A sample of 5 of the possible 34 report topic 
responses was assessed, the same 5 per each report (topics 9A-E). Accuracy, breadth of data items 
considered, and level of detail were the main criteria for comparison. A point system was devised 
for measuring these criteria: as long as a response accurately included most relevant data items, it 
was considered entirely or approximately equivalent to the reviewer’s comparison sample (and thus, 
representative of the data being measured); if a response accurately included 30-67% of data items, 
it was rated partially equivalent to the reviewer’s sample (i.e., partially representative); a response 
including fewer than 30% of the data items was considered barely equivalent (i.e., barely 
representative) . 
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SUMMARIES BY FIELD-TEST SCHOOL 

This section provides a synopsis of each field-test school’ s experience with the School Study 
of Writing Instruction. For the expanded findings, refer to Appendix I. 

Condition 1: Facilitated Schools 

The 5 schools described next were each assigned an external facilitator fi'om the collaborative 
research team to guide them through the School Study process. 

Facilitated School #1 

FS 1 was motivated to conduct the School Study and achieved some successful results. The 
school spent 9 weeks on the study, about twice the amount of time projected, and Steering 
Committee members each spent approximately 1 0 hours, reporting that interviewing and report 
writing took the most time. The facilitator observed that Steering Committee members prepared well 
for their meetings and tasks throughout the study process. Two members of the Steering Committee, 
the assistant principal and a parent, plus another parent not from the Steering Committee, conducted 
interviews; some teacher surveys were employed and students were interviewed mainly in groups. 
Also, FS 1 included individuals fi'om its feeder school in its interview sample. The Steering 
Committee chose to write the report together in a meeting rather than work separately. For the rating 
meeting, the Committee developed a reflection form for gathering faculty feedback about the study 
process. According to responses on this form, faculty learned that teachers and students in the school 
had different understandings of the writing program. Two Steering Committee members echoed this 
finding, saying that this realization was the most helpful aspect of the study process. Faculty also 
reported learning that assistance for and collaboration with content area teachers outside language 
arts were pressing needs, and identified several other needs as well. 

Facilitated School #2 

FS2 completed its study in 5 weeks. Overall, FS2 reported having a negative experience with 
the School Study. The principal did not participate, and the Steering Committee chair reluctantly 
accepted the position of chair. The Steering Committee chair ended up doing a substantial amount 
of the interviewing, report, and compilation work (40+ hours total). Neither did the encouragement 
of the Highly Skilled Educator (HSE) seem to fortify the Steering Committee’s motivation. Despite 
the availability of an external facilitator, the Steering Committee did not contact her for assistance 
beyond what services were offered. The purpose for doing the school study did not appear to be 
understood by the Steering Committee or faculty. The Steering Committee and faculty meetings 
began late and in a state of unpreparedness. Members of the Steering Committee reported that they 
would have liked a longer period of time (and more resources) in which to carry it out. Some 
teachers were interviewed via survey rather than via verbal interview. Various Steering Committee 
and faculty members’ assessments of the teacher and student interviews were that teachers’ 
responses were generally not representative, while students’ responses were. The majority of 
Steering Committee members felt that the school study was a waste of time. The principal’s 
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feedb?L.k stated that the study was worth the effort, in marked contrast to the majority opinion of the 
Steering Committee. 

Facilitated School #3 

FS3 completed its school study in 6 weeks with some successful results. The study 
commenced at the prodding and initiative of the facilitator and in some confusion. The school 
principal had formed the Steering Committee without informing designated members of its purpose; 
however, once the Steering Committee understood and undertook its work, the study gained 
momentum and organization. Two drawbacks to the Steering Committee’s progress were that (1) 
the principal did not participate and (2) two Steering Committee members did most of the work (due 
to a combination of their own volition and perceived expectations of them). The Steering Committee 
never contacted the facilitator beyond what services were offered; the facilitator initiated all contacts 
and assistance. The facilitator was involved in some way at every step of the study process, 
including conducting a few interviews and facilitating the rating session. The Steering Committee 
was reluctant to engage interviewers from outside the school, so 2 of the members conducted the 
bulk of the interviews while the facilitator performed the remainder. They found that interviewing 
students in groups of 2-3 worked well; however, it was sometimes necessary to rephrase questions 
so that students could understand them. Because the faculty in the rating session believed that the 
interviews and report did not accurately reflect their writing program, they had difficulty reaching 
consensus. The facilitator reported that, nevertheless, the discussion produced some shared 
understandings. 

Facilitated School #4 

FS4 completed its school study in 5 weeks. Although FS4 experienced a delay of several 
weeks in beginning the study, the school completed it with some successful results. The principal 
participated actively on the Steering Committee; however, the study began in some disarray. Not 
all individuals had been notified of their designation to the Steering Committee or informed of the 
study’s purpose before the first meeting. The Steering Committee was composed of 9 or 10 
individuals, including at least one parent. The Steering Committee continued to exhibit some 
disorganization; no meeting was attended by all members and the Committee had to rush to finish 
by the end of the year given their delay in beginning the process. The facilitator was integral to 
starting the study on its course and active in several of the tasks involved. Several parents served 
as interviewers. One of the co-chairs reported that students sometimes had difficulty understanding 
interview questions. At the rating session, some faculty members expressed concern that parents 
perhaps were not the best choice as interviewers because they were not sufficiently knowledgeable 
of writing program jargon and, therefore, did not know when to probe students for more precise 
responses. Another obstacle in the interviewing task was that the district administrator refused to 
be interviewed, instead submitting only brief responses in writing. 

The faculty rating meeting far surpassed the Steering Committee’s expectations in scope. 
After rating, they progressed to discussion and reflections on next steps. The Steering Committee 
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also held a meeting after the rating session to begin step 4 of the study process. Most Steering 
Committee members felt that the outcome of their self-study was worth the effort. Individuals’ time 
spent on the study ranged between 2-18 hours. They listed the rating meeting, interviews, and/or 
report as the most helpful elements of the School Study. Compiling the report and meetings were 
the activities that required the most time. 

Facilitated School #5 

FS5 completed its school study in 7 weeks with some successful results. According to the 
facilitator, the assistant principal was a strong leader and assumed the position of Steering 
Committee chair. The principal also participated, and the Steering Committee and faculty were 
eager to do the self-study. The Steering Committee regularly availed itself of the facilitator’s 
assistance even though Committee members believed that they could have conducted the study 
without such assistance. Nevertheless, they reported that the interviewing step would have been 
difficult to accomplish without additional help from other individuals, specifically finding the time 
needed to interview. Two substitute instructional assistants conducted the interviews as apair. They 
interviewed students on sn. individual basis. Several participants in the study expressed the need for 
more teacher and student interviews for greater representativeness. To write the report, the Steering 
Committee worked together over a period of 3 days. The faculty-wide rating session was conducted 
as an entire group rather than in small group format. 

Three of the 4 Steering Committee participants assessed that the outcome of the School Study 
was worth the effort; one said somewhat. The time spent by individual members ranged from 6-24 
hours, various tasks associated with the interviewing step taking the most time. Despite the time 
required to do the interviewing, they ranked the interviews as the most helpful aspect of the study. 
The Steering Committee planned to involve the entire faculty in setting priorities for the writing 
program, although they postponed the meeting plaimed for the end of the school year to a 
professional development session before the start of the following school year. The facilitator 
predicted success for FS5 in following through with the study given the Steering Committee’s 
promising expressions of intent and outlook. 

Condition 2: Independent Schools 

The 6 schools described next were each assigned a contact person from the collaborative 
research team to answer questions about ihQ School Study process via phone or e-mail. The intention 
was for contact persons to be less involved with the study process than the facilitators would be with 
the facilitated schools’ process. 

Independent School #1 

ISl completed its school study in 6 weeks with some successful results. The principal 
chaired the Steering Committee and facilitated the study process. At the committee’s first meeting, 
he presented the School Study handbook to the committee. He called the contact person once with 
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a few questions; she initiated other calls to monitor the school’s progress. The committee chose a 
parent to be the interviewer. The interviewers initially interviewed students on an individual basis, 
but switched to groups of 2-3, which they reported was preferable. The principal and 2 of the 3 
Steering Committee members felt that the school study was worth the effort. The principal was 
appreciative of the opportunity to do it. Steering Committee members, aside from the principal, each 
spent between TA to 1 2 hours on the study, report writing being the task that required the most time. 
As a result of the rating meeting, IS 1 plarmed to increase the participation of non-accountable grade 
teachers in the writing program. The principal expected that they would continue with the action 
plan in the next school year. One member expounded on a recommendation for revamping the 
format of the rating session, saying that it should have more closely emulated a workshop in which 
teachers modeled writing and left with a specific set of actions to implement or expect regarding the 
writing program. 

Independent School #2 

152 completed its school study in 6 weeks with some successful results. The principal was 
eager to do the study and reported being pleased throughout the process. The contact person initiated 
all calls to the school but one that came from the Steering Committee chair. Participants at IS2 had 
few questions. The Steering Committee was composed of 5 or 6 members and the writing cluster 
leader from the district office served as the chairperson. Due to the time constraints of others on the 
Committee, the chair organized the study and carried out most of the tasks herself. The chair 
interviewed students in groups of 4. To save time, teacher interviews were administered as written 
surveys. The Committee incorporated interviews of students and teachers from their feeder school 
in their data collection. Steering Committee members worked separately on their assigned report 
sections. Nonparticipation of non-language arts, content area teachers in the writing program 
emerged as the salient weakness in their program. The Committee convened a second faculty 
meeting for accomplishing the first part of step 4, setting priorities. Responsibility for the second 
part, planning actions, was given to a committee to be composed of the original Steering Committee 
plus a few other individuals. Before the end of the school year, the committee had incorporated new 
actions into their Consolidated Plan and hired a writing specialist. They agreed to complete plarming 
during the summer before the start of the upcoming school year. 

The principal and 2 of the 3 Steering Committee members who responded to the feedback 
form believed that the outcome of their school study was worth the effort. They found the interviews 
to be the most helpful aspect of the study process. Steering Committee members spent from 3 'A - 1 0 
hours to complete the study (although the chair’s estimate seems to be low given the amount of work 
she did), report writing being the most time-consuming task. 

Independent School #3 

153 did not complete the school study to the knowledge of the research team. Initially, it 
seemed underway and was scheduled to be finished by April 5. The contact person first called the 
school in January and called regularly thereafter to check its progress; she also visited the school 
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twice and offered to help at various times. She initiated all contacts with the school. The assistant 
principal assured the contact that IS3 could progress unaided. There were apparently several 
extenuating circumstances that hindered the study’s completion, including dissension from the 
SBDM Council. 

Independent School #4 

154 conducted its school study in 6 weeks with some successful results, although it had a 
slow start due to state assessment and faculty availability. The contact person spoke regularly to the 
principal before the study began; a few subsequent calls were not returned so she did not stay 
updated with the study’s progress. The contact person and Steering Committee chair did not speak 
at any time, so the school conducted its study rather independently. The Steering Committee chair 
replacement was given the position late in the school year. Therefore, she hurriedly organized and 
orchestrated a substantial amount of the early work herself, having the completed interviews ready 
for the other Steering Committee members at their first plaiming meeting. Staff from the Family 
Resource Center interviewed teachers; higher grade level students from another school interviewed 
students; and the chair interviewed the administrators. The chair reported concerns regarding the 
student interviews such as interviewers not probing and interviewees not comprehending the 
questions. Faculty attended the rating meeting on a voluntary basis since it was held near the year’s 
end; slightly more than 50% of the faculty participated. The session was productive, according to 
the chair; they had time for discussion, but not for completing the school profile. The noticeable 
effect of the study was heightened awareness by nonaccountable teachers of the need for them to 
participate in writing instruction. Completing the profile and setting priorities were slated for the 
following school year due to lack of time. 

The principal and 2 of the 3 Steering Committee members believed that the school study was 
worth the effort. One was yet unsure. Committee members spent 6-11 hours each on the study, 
report writing taking the most time. They found the interviews and faculty discussion to be the most 
helpful aspects of the study. 

Independent School #5 

155 conducted its school study in 6 weeks late in the spring with some successful results, 
including germinal beginnings of a relationship with IS5's feeder school. Before the study began, 
the contact person’s role changed to a facilitator’s role by the principal’s request. The 
contact/facilitator reported that the Steering Committee chair was initially uncertain of the study 
process and needed considerable assistance to get started. Also, the chair requested that the 
contact/facilitator perform all of the interviews because the Committee trusted her and knew that she 
was familiar with writing program jargon. Interviews from teachers and students at IS5's feeder 
school were included in IS5's data collection. The facilitator and scribe performed an unusually 
large number of student interviews (21) compared to the handbook recommendation (10-14). 
Moreover they interviewed most students on an individual basis because the scribe conjectured that 
students would respond more openly in that format. The report was scheduled to be written in a 
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group forum to include several teachers from both schools; however, the chair wrote most of the 
report herself The faculty rating meeting was planned for only faculty of the grades encompassed 
by the study from the 2 schools, instead of the entire faculty of IS5. Again, participation was a 
problem in that only 2 Steering Committee members besides the chair attended. By this time, 
several Steering Committee members had quit. 

According to the chair, the study results verified what they had always known to be their 
program’s main weakness — inadequate writing instruction in nonaccountable grades; thus the study 
gave them the evidence they needed to enlist the participation of these teachers to create continuity 
in the writing program. The rating group intended to use its report of preliminary priorities as a 
“springboard” for joint discussion among teachers of certain grades from the two schools at the start 
of the next school year, hoping ultimately for regular joint planning time to align the curriculum. 
The principal and 3 Steering Committee members were unanimous that the school study was worth 
the effort. Steering Committee members invested 4-34 hours each in conducting the study, the chair 
doing much of the work herself The report-writing and rating tasks required the most time, and the 
interviews and dialogue at the rating meeting were the most helpful aspects of the study. Although 
step 4 of the study, setting priorities and planning actions, would not be finalized until the fall of the 
next year, the Steering Committee took initiative in engaging students in a promotional writing 
project during the summer. 

Independent School #6 

IS6 conducted its school study in 9 weeks with some successful results. The Steering 
Committee requested clarification on one issue during the course of the study. The Steering 
Committee included 5 members, who also served as the interviewers and were allotted 1 month to 
complete the task of interviewing. A problem arose in interviewing students, because one of the 
interviewers gave the interview form to a few students to complete in written survey form; these 
forms contained many non-responses. Also, the chair was disappointed with the depth of the other 
student interviews. Steering Committee members worked separately on their assigned report 
sections, each allotted 1 week for this task. One third of the faculty attended the rating meeting, 
which was concluded within 2 hours. Faculty did not engage in extended discussion after the rating 
activity was completed. Step 4 of the study was assigned to two committees in succession — first to 
the school’s writing committee, whose recommendations would be forwarded to the Consolidated 
Planning Committee for finalization. The principal and Steering Committee members all felt that 
their school study was worth the effort. The time investment of individual Steering Committee 
members ranged widely from 6-50 hours, interviewing requiring the most time of members. They 
also found the interviews to be the most helpful aspect of the study. 
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RESULTS 

The results are reported below relative to mostly the 10 field-test schools that completed the 
School Study. One school, IS3, did not complete a self-study; data concerning this school are 
discussed when available. Where appropriate, results are compared between facilitated schools and 
independent schools. 

Initiation of the Study Process and Steering Committee Setup 

Of the 1 1 schools that agreed to participate in the field test and began the School Study, 10 
progressed to a logical interim point of completion by the close of the school year (usually step 3, 
the faculty rating meeting) and 1 school did not follow through in conducting the study. 

The 10 schools that finished, even the schools in which a school administrator expressed 
eagerness in getting started, typically experienced a lapse of 4-12 weeks between the first contact 
with the facilitator or contact person and the first meeting of the Steering Committee, the average 
lapse being slightly over 7 weeks {SD 2.7). The delays were due to postponements by some schools 
and, less often, to the dilemma of identifying a synchronous date between the facilitator/contact 
persons’ and schools’ schedules. Individuals at 7 schools complained that the timing of the school 
study was poor (the spring) because they were busy scoring portfolios and conducting other 
assessments. Although all schools were equipped to begin the school study as early as the first week 
in February, only 2 began by the last week in February; had they started sooner in the spring, schools 
most likely would not have encountered a conflict with timing. Three schools — FSl, IS2, and 
IS 5 — expanded their study to include interviews of students and/or teachers of their feeder school 
in their data collection. 

The Steering Committees of the field-test schools included 4-10 members to start, averaging 
5.5 members per Committee (see Table 2). The facilitators of the 2 schools with a Committee of 8 
or more, FS2 and FS4, thought that this number was too large. Also, facilitators or Steering 
Committee chairs of at least 3 schools reported that some members were not especially interested 
in participating or volunteering for tasks. In fact, 3 schools’ Committees were formed and members 
designated by the principal without designees being adequately invited, or even apprised of their 
appointment and the study’s purpose, until they were summoned to their first meeting with the 
facilitator. At least 7 of the Committees experienced some degree of attrition between the study’s 
beginning and end due to work or personal conflicts; however, at least 3 acquired replacement 
members. 
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Table 2: Selected Aspects of School Study Setup by Field-Test School 





FSl 


FS2 


FS3 


FS4 


FS5 


ISl 


IS2 


IS4 


IS5 


IS6 


1 . Original # Steering 
Committee Members 


5 


8 


5 


9 or 10 


4 


4 


5 or 6 


4 


6 


5 


2. Some Steering Committee 
members chosen from role 
groups besides writing- 
accountable school staff 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


/ 


3. # Steering Committee 

planning meetings 


3 


2 


3 


4 


3 or 4 


4 


2 


1 


1 


2 



Individuals who comprised the Steering Committees represented a variety of role groups, the 
majority being school faculty or administrators. At 5 of the 1 0 field-test schools that completed the study, 
either an assistant principal or principal was actively involved in the study as chair of the Steering 
Committee; at 3 schools, the principal was supportive, but not involved. At the remaining 2 schools that 
completed the field test, the principal was not involved or demonstrably supportive of the study process. 
All Committees included faculty members, 7 of which engaged writing nonaccountable teachers (those 
teaching in content areas other than language arts of any grade or in language arts outside of the 4"’, 7"’ and 
12* grades) as well as writing accountable .teachers (language arts teachers of the 4*, 7*, and 12* grades). 
Of these 7 schools, 5 also engaged special staff such as the guidance counselor. Title I teacher, or 
instructional aides. At least 6 of the 10 schools invited parents to participate on the Steering Committee. 
Finally, 2 schools involved a unique member on their Committee — IS2 included its district writing 
specialist and IS6 included a recent graduate who was studying education in college. 

Steering Committees held between 1 and 4 planning meetings in the course of the school study 
(excluding meetings in which they actually accomplished a step in the process, such as report writing or 
rating), the average being 2.6 meetings. 

The Interviews (Step 1) 

Almost all field-test schools (9) enlisted two or more interviewers/scribes to conduct interviews (see 
Table 3). Only 1 school, IS2, employed a sole interviewer. Steering Committees chose interviewers from 
a variety of role groups, but often from within the Committee itself instead of from outside the school as 
recommended. At 6 schools, some or all interviewers were administrator or faculty members of the 
Steering Committee; they were primary interviewers^ in 5 of these 6 schools (see Table 3). At 1 school, 
interviewers were school staff (instructional aides) not on the Steering Committee. At 5 schools, some or 



*In this usage, primary interviewer means someone who interviewed teachers and/or students. An administrator or 
faculty member of the Steering Committee who interviewed only school or district administrators would not be considered a 
primary interviewer (i.e., as in 1 school, IS4). 
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all interviewers were parent members of the Steering Committee or other parents. At 3 schools, perhaps 
4, the facilitator or contact person, or another individual in a facilitative role such as a Highly Skilled 
Educator (HSE), helped with interviews. At 1 school, IS4, some of the interviewers were Family Resource 
Center staff and some were students from a neighboring school, the latter group being an unconventional 
yet suggested choice in the School Study handbook. However, the IS4 Steering Committee chair reported 
that the interviews conducted by students were not successful. Six schools employed interviewers from 
a combination of these various role groups. 

In sum, half of Steering Committees reserved interviewing partially or wholly for themselves or 
school staff, which was recommended against in the handbook except for administrator interviews. Those 
associated with the school in an external capacity, such as parents, were acceptable choices; however, it 
was not intended for facilitators/contact persons to perform this function. In 2 schools, the facilitator’s 
interviewing assistance was minimal, confined to administrator interviews; however, in 1 independent 
school, IS5, the contact person performed all teacher and student interviews aided by a parent. 



Table 3; Selected Aspects of Interviewing (Step 1) by Field-Test School 





FSl 


FSl 


FS3 


FS4 


FS5 


ISl 


ISl 


IS4 


IS5 


IS6 


1. 


# Interviewers/Scribes 


3 


3 


3 


5 


2 


2 


1 


6 


2 


5 


2. 


Primary interviewers 
chosen only from role 
groups external to the 
school 








/ 




/ 


/ 


/ 


/ 




3. 


Time frame for 
interviewing (in days, 
except wks=weeks) 


2 


2 


3 wks 


7 


2 


10 


2 


7 


7 


4 wks 


4. 


Students interviewed 
in groups 








some 




some 











Four schools accomplished the interviews within 2 consecutive days (see Table 3). Four other 
schools reported spending a few days time over a period of 7-10 days. Two schools spread the 
interviewing task over 3-4 weeks. The length of one interview was a typical 45-60 minute class period for 
both teachers and students, whether in group or individual interviews. 

Schools interviewed from 4-12 teachers depending on the school population, in keeping with the 
handbook guidelines. At least 4 schools interviewed more teachers than suggested. Four schools solicited 
some or all teacher interviews via written survey rather than verbal interview . The Steering Committee 
chair at FS2 interviewed teachers in pairs, which was counterproductive for eliciting the most candid 
replies; she later complained of the unrepresentativeness of teacher responses. 
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Of 7 reporting schools, all interviewed more students than suggested in the handbook given their 
student population. Six schools interviewed 10-12 students and 1 school, IS5, interviewed a sizable 
sample of 21 students. 

Interviewers at 4 schools interviewed students in groups of 2-4 (see Table 3). Interviewers at 2 other 
schools interviewed some students in groups and some on an individual basis. At 3 schools, interviewers 
conducted almost all student interviews on an individual basis. One school, IS6, administered some of 
their student interviews via written survey, which did not work well because students answered briefly and 
left many items blank. The interviewers’ preconceptions of what format was most advantageous for 
engaging student interviewees determined whether they conducted group or individual interviews. Those 
who believed that group interviews would be preferable were borne out. Those who opted for individual 
interviews either found them to be ineffective or, to save time, switched to groups and found this method 
to be preferable. Written surveys were never a suggested option for student interviewees, and IS6's 
experience amply illustrated why. 

It was not always clear who interviewed the principals and district administrators of the field-test 
schools. In some cases, the designated interviewer did and in others, the interviewees completed written 
surveys. At FS3 and FS4 in particular, each facilitator was asked to interview the district administrator 
because Steering Committee members expected uncooperativeness that they wished to personally avoid; 
in both cases, the district administrator refused to be interviewed but returned a written survey. 

Only 5 schools typed some of the completed interviews, none typing all of them; of these, only 3 
typed compilations of either student and/or teacher interview responses. Four gave estimates of typing 
time; 1 hour, I'A days, 4 days, and 1 week. Only 2 schools, FS3 and IS5, enlisted a helper not on the 
Committee for typing and compiling. At the 3 other schools with typed interviews or compilations, either 
the Steering Committee chairs or, in one case, the parent interviewer did them. 



Revision to the School Study handbook. The use of written surveys as a substitute for verbal interviews of 
teachers was intended to be a last resort option only. Yet it seems that interviewers frequently relied on teacher 
surveys and, erroneously though less often, student surveys. Therefore, the language that presents the teacher 
survey as an option will be removed from the final version of the School Study handbook. Although verbal 
interviews may not permit interviewees to deliberate over their responses as thoroughly, they save time and 
add an important interactive element to the study process. Also, verbal interviews produce briefer responses, 
making it easier for the Steering Committee to write the report. This change to the handbook is reflected in 
Table 13 at the end of the Results. 



The Report (Step 2) 

Schools adopted various forums for writing the school report (see Table 4). Four Steering 
Committees divided the report among themselves and completed their assigned sections individually. 
None reported working in pairs. Four Steering Committees worked together in a group meeting or series 
of meetings, an interesting variation because it was not presented in the handbook as a possibility. Finally, 
at 2 schools, FS3 and IS5, 1 or 2 Steering Committee members took responsibility for writing the entire 
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report themselves; this was not recommended, although at IS5, the chairperson had no choice because 
others did not show up at what was supposed to have been a group undertaking. 



Table 4: Selected Aspects of Report Writing (Step 2) by Field-Test School 





FSl 


FSl 


FS3 


FS4 


FS5 


ISl 


ISl 


IS4 


IS5 


IS6 


1 . Format for report writing 
[mtg=meeting(s) ; 
mdiv=individually; 
pers=person(s)] 


mtg 


mtg 


indiv 
(1 pers) 


mtg 


3 mtg 


indiv 


indiv 


indiv 


indiv 
(1-2 pers) 


indiv 


2. Range of individuals’ time 
spent on report writing 
(in hours, ex. wk=week) 


3-8 


3-25 


3-10 


2-4 


12 


2 


2-5 


2-4 


2-10 


1 wk 



Report writing was most frequently identified by Steering Committee members as the most time- 
consuming task of the school study process. Time estimates for report writing came from two 
sources — Steering Committee chairs in phone interviews and/or individual Steering Committee members 
in feedback forms. Estimates were provided either as individual hours spent per Steering Committee 
member; as hours spent by the collective Steering Committee in a group meeting or series of meetings, 
which would apply equally to each participating member; or as a time period (1 school, IS6, said 1 week). 
Respondents from nine schools provided estimates in hours, giving a total of 24 estimates (see Table 4).'° 

Of the 24 time estimates, the range was 1.5- 22.5 hours per person; the most frequent response was 
3 hours; and the average was 5 hours. If the highest estimate were discarded (2214 hours), the range would 
be 1.5 -12 hours and the average would be 4.1 hours. Thus, the average time spent per individual to 
contribute to the report may be placed somewhere between 4 and 5 hours. A few of these estimates include 
contemporaneous typing time, but most do not. 

All schools typed their report. Time estimates given for typing only, done separately from report 
writing, ranged from 1-10 hours per individual. Only 5 schools enlisted a helper for the typing task; except 
for possibly FS4, which may have requested a parent’s volunteer assistance for typing, these schools asked 
paid staff such as an administrative or instructional assistant to type the report. At the other 5 schools, 
either the Steering Committee chair typed the entire report or individual Steering Committee members 
typed their own sections. 



'^However, these should not be considered estimates from 24 separate individuals, since a Steering Committee chair 
may have provided an estimate to apply to all Steering Committee members, and some of those same members may have given 
their own personal estimate in a feedback form; thus there are two estimates for some individuals. Yet they are all counted in 
this examination to provide as representative an estimate as possible. 
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The Rating Meeting (Step 3) 

Almost all field-test schools structured the rating meeting the same (see Table 5). Eight schools 
divided faculty among small groups as prescribed in the handbook. However, 2 schools adopted a different 
structure for their meeting: FS5 completed its ratings as a whole faculty group and IS5 targeted only the 
specific teachers who taught the grade levels under scrutiny in IS5's study. Of the 10 schools, 3 
schools — IS4, IS5, and IS6 — invited faculty on a voluntary basis and consequently reported low turnouts 
(approximately 33-50%), even though IS6 offered stipends and credit toward professional development 
requirements. The main detractor from attendance at IS5 was that the rating meeting was held after school 
had closed for the year (in addition to having invited only a select group of teachers). Of the other 7 
schools, 5 reported attendance at 70-100% and 2 did not report a figure. 



Table 5: Selected Aspects of Rating (Step 3) by Field-Test School 





FSl 


FSl 


FS3 


FS4 


FS5 


ISl 


ISl 


IS4 


IS5 


IS6 


1. Faculty in small groups 
format for rating meeting 


/ 


/ 


/ 


/ 




/ 


/ 


/ 




/ 


2. Rating meeting attendance 
satisfactory 


yes 


? 


yes 


? 


yes 


yes 


yes 


no 


no 


no 



The time span for schools’ rating meeting ranged from 1-3 hours (except for IS5's rating meeting, 
which was an intensive 6-hour session among 3 people that incorporated a substantial amount of planning). 
Five schools estimated approximately 2 hours, and the average for all schools except IS 5 was 2 hours. In 
phone interviews or other means of feedback, most Steering Committee chairs commented that the rating 
meeting was expeditious, concluding much earlier than expected. According to the chairperson, even the 
3 individuals at IS5 whose meeting was quite long (6 hours) felt that it was a wonderful experience. 

The length of each school’s meeting varied somewhat depending on whether the faculty progressed 
to preliminary work on step 4 by discussing priorities after completing the rating activity. For instance, 
IS2's meeting, which lasted only 1 hour, adjourned immediately after rating with no follow-up discussion. 
Yet 9 of the 10 schools progressed to discussion afterwards, which led to preliminary priority work in 8 
schools. 

Facilitators and Steering Committee chairs of some schools reported initial impediments to 
discussion such as disinterest, inflated perception of the writing program’s current effectiveness, or 
reluctance to voice controversial viewpoints; yet all those who reported these hindrances added that useful 
discussion evolved nevertheless. On the other hand, a principal and Steering Committee chair at two 
schools, FS 1 and FS3, viewed their discussions as tedious or indicative of a problem in their study process 
(rather than a means of furthering understanding) because the faculty had difficulty reaching consensus. 
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A few schools shared a unique aspect about their rating meeting. FSl asked rating meeting 
participants to reflect on what they had learned on a 2-item questionnaire at the meeting’s conclusion. At 
ISl, one Steering Committee member criticized that the rating meeting was not extensive enough for 
motivating teachers; this individual proposed a different orientation for the rating meeting that would more 
effectively stimulate teachers’ planning and responsibility for improvement. 

Priorities and Planning (Step 4) 

Steering Committees of the 1 0 schools orchestrated step 4 in a variety of ways (see Table 6). Seven 
schools assigned responsibility for priorities and planning to a committee: the Consolidated Planning 
Committee, the Writing Committee (either a separate entity or part of the former), the SBDM Council, a 
committee to be newly formed for the purpose, or the original school study Steering Committee. Of these 
7 schools, 3 divided the step 4 activities between two of these committees or between the entire faculty 
and one committee. Two schools — FS5 and ISl — assigned responsibility for step 4 to the entire faculty, 
rather than to a committee or some combination of such. One school did not report its structure for step 
4 because it had not plaimed it yet. 



Table 6: Selected Aspects of Priority-Setting and Planning (Step 4) by Field-Test School 





FSl 


FS2 


FS3 


FS4 


FS5 


ISl 


IS2 


IS4 


IS5 IS6 


1. 


Format for priority-setting 
and planning (com= 
committee(s); fac=faculty) 


com 


com 


com 


2 com 


fac 


fac 


fac & 
com 


none 

yet 


com 2 com 


2. 


Progression to priority- 
setting (or preliminary 
discussion) 








/ 




/ 


/ 




/ 


3. 


Progression to planning 








/ 




/ 


/ 




/ 


4. 


Progression to actions 














/ 




/ 



In review, it seems that some schools assigned step 4 responsibilities so as to preserve the integrity 
of progress already made in their school study. That is, they would assign at least part of step 4 to a 
decision-making body that was already somewhat involved with the study — such as the Steering 
Committee, Writing Committee, or whole faculty. Whereas if they gave the entire task to a group 
unfamiliar with what was already learned, the new group might regress to needs assessment, essentially 
redoing what had already been accomplished. 

Eight schools engaged in setting priorities, the first phase of step 4, to some extent (see Table 6). 
All 10 schools either reported the intention to finish step 4 activities in the 1999-2000 school year or gave 
no indication that they had completed them by the close of the 1998-1999 year. Although they did not 
report having finished, 4 schools did begin planning actions, the second phase of step 4, by the close of 
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the year— FS4, ISl, IS2, IS5 — a few already incorporating new actions into their Consolidated Plan. Of 
these 4 schools, 2 progressed to acting on one of their plans — IS2 hired a writing specialist and IS 5 
involved its students in a promotional project for Habitat for Humanity over the summer. 



Revision to the School Study handbook. An assessment of how field-test schools commenced and proceeded 
with step 4 activities will has caused a revision in the School Study handbook. The rating meeting (step 3) 
proved to be a good forum for moving to step 4 in that most schools began priorities work during the course 
of the meeting. Thus it was decided that the first part of step 4, setting priorities, would be added to step 3 in 
the final version of the handbook. This modification possesses the benefit of preventing schools from losing 
momentum by pausing too long before completing priorities and planning, as some field-test schools did. The 
second part of step 4, planning actions, instead of remaining as step 4, will be recast as a supplemental section 
in the handbook, thereby providing the indication that planning should follow but making the study process 
smaller by not presenting planning as an entirely new step. These changes in design should make the School 
Study process less formidable overall. They are listed in Table 13 at the end of the Results. 



Assessment of the Steps and Products of the School Study 

Comparative time requirement per step of the School Study. Best estimates for each of the 
study’s steps are provided in Table 7 for a convenient review and comparison. 



Table 7 : Time Estimates per Step of the School Study of Writing Instruction 



Orchestration & 
clerical tasks: 


Steering Committee planning 
meetings and preparations 
Compiling/typing interviews 
Typing the report 
Various copying 


4-7 hours per Committee member 
14-21 hours (over 3 '/a days) 

3-10 hours 

no estimate provided by the data 


Step 1: 


Interviewing 


14 hours (over 2-3 days) 


Step 2: 


Writing the report 


3-5 hours per Committee member 


Step 3: 


Rating (+ some prioritizing) 


2 hours per faculty participant 


Step 4: 


Setting priorities 
Planning actions 


1 hour per participant 
no estimate 



Of the 43 Steering Committee members who completed a feedback form, all replied to question 2, 
which tasks in the School Study required the most time, with a total of 66 responses. Compiling the report 
was the most frequent answer, meriting 28 responses. Several of these respondents also said that, in 
addition to the actual writing, reading through interview responses prior to writing was time-consuming. 
Next were tasks associated with Steering Committee planning and preparation, such as attending planning 
meetings, reading the handbook, scheduling interviews, copying interview forms, and typing compilations 
of teacher and student interviews ( 1 8 responses). The third most frequent response was interviewing ( 1 3 
responses); all of these responses came from the 5 schools at which Steering Committee faculty or other 
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faculty members conducted some or all of the interviews themselves. The fourth response was the rating 
meeting (5 responses). The last 2 responses were “other” and one that was incomprehensible. For a 
comparison of responses by school, see Table 8. 



Table 8: Time Requirement and Helpfulness of School Study Steps by Field-Test School 





FSl 


FS2 


FS3 


FS4 


FS5 


ISl 


IS2 


IS4 


IS5 


IS6 


1. Step(s) that required the 
most time 

(report==report writing; 
intvw=interviewing) 


report 


intvw; 

report 


intvw 


report 


intvw; 

report 


report 


report 


report 


rating 


intvw 


2. Most helpful parts/steps 
(intvw=interviews) 


intvw 


none 


intvw 


intvw; 

report 


intvw 


intvw 


intvw 


intvw 


intvw 


intvw 



Most helpful parts of the study. Question 8 on the Steering Committee feedback form asked 
which parts of the study process were most helpful. Forty of the 43 Steering Committee participants who 
completed a feedback form responded to this question; 3 did not because their form was missing the 
question. The 40 respondents gave a total of 50 answers. By an overwhelming margin, the most frequent 
response was the interviews (26 responses). Several respondents specified either the student or teacher 
interviews or both, either because they judged them to be honest or judged them to be inaccurate. Several 
referred to the comparison of discrepant viewpoints among role groups. The second most frequent response 
was the rating meeting discussion or the school profile generated in the rating meeting (9 responses). The 
third highest response was the report (5 responses). The remaining 10 responses included the Steering 
Committee’s collaborative work, the rating guide, heightened understanding or awareness of needs, 
everything, and nothing; the 4 responses of “nothing” all came from respondents at FS2. For a comparison 
of responses by school, see Table 8. 

Least helpful parts of the study. Question 9 on the Steering Committee feedback form asked 
which parts of the study process were least helpful. Thirty-two of the possible 43 respondents answered 
this question; 8 did not answer; and 3 did not answer because their form was missing the question. The 
32 respondents gave a total of 35 answers. Seventeen responded that no parts of the study were least 
helpful." From the remaining 15 respondents who specified an aspect(s) of the study, 6 responses referred 
to errors in the School Study instrument; 5 responses referred to either teacher or student interviews; 5 
referred to one of the following aspects — the time frame, the tedium of the rating meeting, the lack of 
parent and community involvement, or the fact that it was a review of what was already known; and 
finally, 2 responses were that all parts were least helpful, both of these responses coming from individuals 
atFS2. 



" If the 8 nonresponses are interpreted as indications that these 8 nonrespondents likewise felt that no parts of the study 
were least helpful, and are added to the 17 who responded as such, the modified number of those who felt that no parts were 
least helpful would be 25. 
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Assessment of the represenlativeness of the school report. The report produced from step 2 of 
the study process was examined for reliability for 7 of the 10 field-test schools that completed the self- 
study. For 3 of the schools, an authentic examination could not be performed. The report was chosen as 
the document for the external review (as opposed to the rating form or school profile) because it serves as 
the synthesis of the raw interview data and then the source document for rating the writing program, 
making it the key information valve in the study process. 

In terms of accuracy and breadth of data points considered, 4 of the 7 schools’ reports were entirely 
or approximately equivalent to the reviewer’s version in most of their responses.'^ Two schools’ reports 
were at least partially equivalent in most of their responses. One school’s report was barely or not at all 
equivalent in most of its responses. Although these results are favorable overall, most reports were not 
comprehensive but brief with a lack of detail: only 2 incorporated substantial and strategic detail, another 
2 did so partially, and 3 did not at all. Additionally, the “Other” space was used in very few reports for 
recording additional information not gleaned from the interviews; however, sometimes this was due to 
writers including such commentary in the designated role group spaces along with the other data. 

Also, schools’ reports sometimes contained misrepresentations; usually only one such error was 
noted per school. The reviewer noticed that a misrepresentation often occurred in the same report topic 
{9C— student opportunity to compare writing between school years, which is essentially a question about 
the number of grade levels at which portfolios are required of students). Report writers would say that 
portfolios are required by teachers of all grades, when actually, the interview data did not show that to be 
the case. 

Assessment of the representativeness of the school profile. Participants were asked to assess the 
value of another product of their school study — the school profile produced in step 3 of the study process 
(the rating meeting). Both principals and Steering Committee members were asked whether the school 
profile was an accurate picture of their writing program ’s strengths and weaknesses (question 1 on the 
principal form and question 5 on the Steering Committee form). In a second part to the question, they were 
asked what about the study process may have caused inaccuracies, if any. 

Out of a total of 5 1 principals and Steering Committee members who completed a feedback form,'^ 
45 answered the question and 6 did not (3 because the question was missing from their form). Of the 45 
respondents, 28 said yes (the profile was accurate), 7 said mostly or somewhat, and 10 said no (5 being 
from 1 school — ^FS2). In 8 of the 10 schools, the majority of respondents felt that the profile was 
somewhat to totally accurate. 



'^Incidentally, school faculties were more stringent in their impression of their school’s report than the external 
reviewer. Although the reviewer gave relatively high reliability assessments to the reports of these 4 schools, faculties at 3 of 
them stated at the rating meetings that the report was unrepresentative of their program. 

'^The principal of FS4 filled out both a principal and Steering Committee form, so there were actually 52 forms total; 
however, his answers are counted only once. 
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Faculty members who were not on the Steering Committee were not asked to complete a feedback 
form. However, their overall impressions of the school study were conveyed by facilitators/contacts and 
Steering Committee chairs: according to these sources, faculties at 3 schools expressed dissension with 
some of the ratings shown on the school profile. (Some dissension was expected as a part of the process 
of discussion and agreeing on ratings.) 

In phone interviews and responses to various questions on the feedback forms (not just to question 
5), study participants identified several aspects of the study that they thought were sources of inaccuracies 
to the report, ratings, or school profile: 

• Several participants from 7 schools felt that the teacher and/or student interviewee samples 
were not large enough (of the 7 schools that made this observation, 3 did interview more 
teachers than called for and 2 interviewed more students). 

• Two individuals named the randomness of the teacher or student interviewee samples as a 
detraction from the accuracy of study findings, one suggesting that teachers be allowed to 
choose the student interviewees. 

• Several individuals from 6 schools evaluated the honesty of teacher and/or student interview 
responses, or expressed an a priori concern about candidness that caused them to orchestrate 
interviews in certain ways. Individuals at 4 of these schools identified less than candid 
responses in either the teacher or student set of interviews, whereas others simply identified 
inaccuracies without a qualitative judgement that they were intentional. In the same vein, 4 
school faculties were skeptical or cynical about the validity of district administrator interviewee 
responses regarding district support of the writing program. 

• Repeated references were made to a concern that particular role groups did not understand 
interview questions or the true meaning of writing well enough to ask or respond correctly. 
These role groups were identified as elementary and middle grades students (mentioned by 
participants at 5 schools); certain interviewers, such as parents or others unfamiliar with writing 
program jargon (mentioned by 3 schools); and teachers (2 schools). 

• Many individuals at 8 schools complained that (a) interview questions were not linked clearly 
to report topics, and/or (b) indicators were not aligned correctly between the report and the 
various rating tools (i.e., rating guide, rating form, school profile), causing them to record data 
or rate indicators insufficiently. 

Suggestions for improvements to the study. Both principals and Steering Committee members 
were asked for suggestions to improve the School Study of Writing Instruction (question 4 on the principal 
form and question 10 on the Steering Committee form). Forty-three of a possible 5 1 respondents gave 55 
responses and 8 did not respond, 6 because their form was missing the question due to a copying error. 




43 



33 



Of the 55 suggestions, 26 were recommendations involving changes to the School Study handbook. 
Twenty of these 26 dealt with an aspect of the interviews, such as aligning interview questions to the 
report, condensing the number of questions, or changing the guidelines for selecting the interviewee 
samples. Two were suggestions to correct the misalignment of indicators on the rating form. Three were 
recommended modifications to the basic design of the study process — incorporate annual reviews and 
restructure the rating meeting. Finally, 1 proposed simplifying handbook guidelines in general. 

Fourteen of the 55 responses were recommendations related to the logistics of conducting the study. 
Five were suggestions to lengthen the time frame. Five related to the study’s timing in the school year. 
Three concerned the need for more financial resources (all from FS2). Finally, 1 was a suggestion that the 
facilitator more fully explain the purpose and benefits of the study to the Steering Committee. 

Of the remaining 15 responses, 10 were statements that no improvements were needed, 2 were 
recommendations to discontinue the School Study altogether (both from FS2), and 3 were 
incomprehensible. 

Section summary. The interviews step was the most frequent response to the four questions on the 
feedback form(s) that asked respondents to reflect on specific aspects of the study process. By a large 
margin, respondents found the interviews to be the most helpful aspect of the process. At the same time, 
they most frequently pinpointed the interviews as the source of inaccuracies in the final ratings; thus, most 
suggestions for improvement related to the interviewing step. Despite these perceived inaccuracies, most 
Steering Committee participants felt that the school profile accurately reflected their school’s writing 
program. 



Revisions to the School Study handbook. Many participants (a) reported having difficulty correlating 
interview questions to the report topics for writing and (b) identified misalignment between indicators on the 
report form and indicators on the rating tools (i.e., rating form and school profile). 

Regarding concern (a), the research team decided that the “Table of report sections and data,” which had been 
used in the pilot test but substantially altered for the field test, would be revised and reinserted in the final 
version handbook to serve as a data-linking guide (see Appendix J). The team also decided to remove the two 
sample reports, one that was very brief and one that was comprehensive, and replace them with a new medium- 
depth one. 

Regarding concern (b), the research team reviewed the handbook and confirmed that some indicators were 
misaligned among tlie various forms. Some later identified indicators, as well as revisions to existing 
indicators, had been added to some forms but inadvertently omitted from others. As a result, not only were 
some indicators missing, but in a few instances the same indicators were numbered and phrased differently 
among the forms. In sum, in the process of correcting these errors, the research team discovered that there were 
36 indicators of successful writing programs, not 34 (see Appendix K). All forms were corrected for the final 
version of the School Study handbook. The changes discussed here are reflected in Table 13 at the end of the 
Results section of this report. 
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Time Span and Time Investment 

Time span. Once schools undertook the School Study, they completed it within a span of 5-9 
weeks, the average completion period being 6.5 weeks {SD 1.4) (see Table 9). These figures represent 
total time spent, but not necessarily a continuous period of time, since some schools suspended their study 
for spring break and/or a state-testing period. There was not a notable difference between facilitated 
schools’ and independent schools’ average completion times. Several independent schools were able to 
complete the study in 6 weeks. The only 2 schools that completed the study in as few as 5 weeks were 
facilitated schools; however, participants at 1 of these schools, FS2, reported feeling rushed. 



Table 9: Time Span of Field-Test Schools to Complete the School Study of Writing Instruction 





Facilitated schools (FSs) 


Independent schools (ISs) 




FSl 


FS2 


FS3 


FS4 


FSS 


ISl 


IS2 


IS3 


IS4 


ISS 


IS6 


Time span in weeks 


9 


5 


6 


5 


7 


6 


6 


— 


6 


6 


9 


Average for FSs & ISs 


6.4 weeks 


6.6 weeks 


Average all schools 


6.5 weeks 



Time investment Forty-three Steering Committee members completed a feedback form, and all 
responded to question 1, what was one’s personal time spent to conduct the School Study. Of the 43 
respondents, the range of hours spent per individual was 2-50 hours; the average per individual was 13.5 
hours {SD 9.1) (see Table 10). For FSs only, the range was 2-40 hours and the average was 13.8 hours {SD 
7.4). For ISs only, the range was 3.5-50 hours and the average was 12.9 hours {SD 1 1.6). 



Table 10: Reported Individual Time Spent to Conduct the School Study of Writing Instruction 



Hours spent 


# Steering 
Committee 
respondents 


Range 


Mean 


SD 


1-10 


20 








10+ - 20 


13 


2-SO 


13.S 


9.1 


20+ - 30 


7 


hours 


hours 


hours 


30+ 


3 








N = 43 respondents 



Several individuals at both facilitated and independent schools spent a substantial number of hours 
to conduct the study (see Table 11). More than 20 hours was arbitrarily chosen to define a substantial 
number of hours. At facilitated schools, 7 of 27 respondents from 3 schools reported spending more than 
20 hours. At independent schools, 3 of 16 respondents from 2 schools reported spending more than 20 
hours. Also, based on other data, a Steering Committee chair from 1 additional IS is believed to have 
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spent this much time but not reported it. In sum, approximately 10 or 1 1 individuals from either 5 or 6 
schools (both FSs and ISs), representing about !4 of Steering Committee respondents, worked a substantial 
number of hours. Moreover, at all schools except one, only 1 or 2 Steering Committee members did so 
compared to their Committee co-members (whereas at FS2, several individuals on the Committee spent 
substantial hours). Of these 6 schools, 3 were ISs and 3 were FSs. 



Table 1 1 : Individuals Investing Substantial Hours in the School Study by Field-Test School 





FSl 


FSl FS3 FS4 FSS ISl 


ISl 


IS4 


ISS 


IS6 


# Steering Committee 
members who invested 
>20 hours 




4 1 2 


? 




1 


2 



Comparison of Facilitator/Contact Person Assistance 
between Facilitated and Independent Schools 

Services provided and not provided. The various services that facilitators and/or contact persons 
provided are listed in Table 1 2. As intended, facilitated schools and independent schools received different 
levels of services to conduct the School Study of Writing Instruction. 



Table 12: Comparison of Facilitator/Contact Person Services 
Used by Facilitated Schools (FSs) and Independent Schools (ISs) 





# FSs that 
Received Service 


# ISs that 
Received Service 


1. Prompting initiation of a school’s study 


3 


0“ 


2. Preparing for Steering Committee meetings 


5 


1 


3. Leading the first meeting and/or introducing the 


5 


1 


School Study handbook 






4. Facilitating or attending subsequent planning meetings 


5 


0 


5. Selecting interviewers and interviewees 


4 


0 


6. Training interviewers 


3 


0 


7, Serving as an interviewer 


2 


1 


8. Facilitating report writing 


4 


0 


9. Facilitating the faculty rating meeting 


3 


0 


10. Facilitating priority-setting and planning 


0 


0 



“This figure includes only those ISs that completed the study. IS3 did require prompting, however, did not complete the study. 



Three FSs required prompting to commence their school study, although 1 of these schools, FS4, 
was clearly motivated to do the study and only needed to be reminded. One independent school, IS3, 
needed considerable prodding yet did not finish the study despite repeated encouragement and offers of 
assistance; none of the 5 ISs that finished the study required prompting that was apparent or reported. 
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All 5 FSs, plus IS5, received facilitator/contact assistance to prepare for Steering Committee and 
faculty meetings. The facilitator at FSS also helped the Steering Committee with copying prior to some 
meetings. 

Again, at all 5 FSs plus IS5, the facilitator/contact was present at the first Steering Committee 
meeting. At FSS and IS5, which had the same facilitator/contact, the facilitator/contact attended a pre- 
meeting with the Steering Committee chair as well as the initial Steering Committee meeting. At 3 of the 
5 FSs, the facilitator introduced the School Study process and handbook. At FS 1 , the Steering Committee 
chair and the facilitator jointly reviewed the process for the rest of the Committee. At only 1 facilitated 
school, FSS, did the chair present the School Study herself At all S ISs that completed the study, the 
Steering Committee chair presented the School Study. 

At all facilitated schools, the facilitator attended subsequent planning meetings and/or work 
meetings (for report writing and rating) of the Steering Committee. Contact persons did not attend 
subsequent meetings of the Steering Committee at any of the independent schools. 

Facilitators/contact persons provided several types of aid related to the interviewing step of the 
study. Again, and as expected, facilitated schools generally received more assistance than independent 
schools. At 4 FSs, the facilitator participated in or guided the selection of interviewees. At 3 FSs, the 
facilitator trained either the Steering Committee or interviewers in interviewing techniques. At 2 FSs, plus 
ISS, the facilitator acted as an interviewer. (It was supposed that facilitated schools might be more inclined 
than independent schools to request interviewing services from their facilitator/contact because they had 
more involvement with them. Yet the 2 FSs requested only minimal facilitator help with interviews, while 
ISS requested a substantial amount of time from its contact person for all interviews.) 

Concerning report writing, the facilitators gave guidance at 4 FSs. At 3 of these schools, the 
facilitator attended a group meeting convened for the purpose of starting or completing the report. At FS3, 
the facilitator gave direction for expanding the report after it had been written. No ISs received facilitation 
for the report- writing step. 

Two aspects of writing the report, type of forum and time spent, were examined in relationship to 
the school’s status as a facilitated school or independent school. Regarding the first variable, a Steering 
Committee’s forum for writing the report, there was a noticeable difference. All facilitated schools except 
FS 3 worked in a group setting while all ISs worked individually on their report sections.*'* (Two of the 
4 FSs that worked as a group chose to do so of their own accord, while 2 did so at the suggestion of the 
facilitator.) Regarding the second variable, time spent to complete the report, there Wcis no perceptible 
difference between facilitated and independent schools. 



*'*IS5 had planned to work in a group meeting, but participants did not show up; thus the chair wrote most of the report 

herself. 
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Concerning the faculty rating meeting, the facilitator led the meeting at the Steering Committees’ 
request at 3 FSs. One additional FS originally intended to have the facilitator present, but did not call her 
after the original date for the meeting was changed. As expected, no rating meetings at ISs were attended 
by contact persons. 

Concerning setting priorities and planning actions, neither facilitators nor contact persons aided FSs 
or ISs, respectively. (However, some facilitators were present at the rating meeting when preliminary 
priority-setting took place.) 

There were a number of other tasks, related to various steps of the study process, for which 
facilitators did not provide assistance. These were substantial copying jobs (of handbooks, forms, and 
completed documents), compiling interviews, writing and distributing the report, and typing. 

Perceived need for assistance. On the whole, field-test schools (both facilitated and independent 
schools) rarely contacted their facilitator or contact person for assistance. Usually, the facilitator or contact 
person initiated phone and e-mail contacts. When schools (principals or chairs) did initiate contacts, they 
did so only once or twice and had only one or two questions; furthermore, they were as likely to call 
someone at one of the sponsoring agencies of the research team as they were to call the facilitator/contact. 
Only 2 schools — ^FS5 and ISS — contacted their facilitator/contact person fairly often. 

Question.4 on the Steering Committee feedback form inquired about respondents’ perceptions of 
their ability to have conducted the School Study without assistance and for what tasks in the study they 
might or did require help. The question was worded differently between the feedback forms given to 
facilitated schools and independent schools. For FSs, the question was (a) whether they could have carried 
out the study without outside help and (b) what tasks would have been difficult without outside help. For 
ISs, the question was (a) at what points in the study did they need to contact someone with questions and 
(b) for which tasks did they need more assistance than available. 

Twenty-seven Steering Committee feedback forms were received from facilitated schools. In 
response to the first part of question 4, 1 5 said yes (they could have carried out the study without outside 
help), 7 said no, 1 said possibly, 3 were unclear, and 1 did not respond. Fourteen of the 27, including those 
who had answered yes and those no, responded to the second part of the question (what tasks would have 
been difficult . . .) with a total of 21 responses. Of the 21 responses, 10 concerned the initial overview and 
setup of the study, 6 related to interviewing, 4 related to the report, and 1 to leading m_eetings. What is 
noteworthy about almost all of the 21 responses is that they were the specific activities for which each 
school had actually received assistance from the facilitator. Virtually none related to activities for which 
respondents had not received help; only 1 school, FSS, listed a task with which it was unaided but would 
have appreciated outside help (interviewing). Thus, almost none claimed needing more assistance than 
that already supplied to them. 

Sixteen Steering Committee feedback forms were received from independent schools. In response 
to the first part of question four, 8 of the 16 respondents reported contacting someone with questions 
during the study, at least 3 of whom consulted another individual in the school rather than the contact 
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person. Five of the 1 6 respondents reported having no need to contact someone with questions, 3 adding 
that their questions were answered amply by their Steering Committee chair. Three of the 16 did not 
respond. In response to the second part of question 4 (for which tasks did they feel the need for more 
assistance than was available), all 16 said none or gave no response, 2 adding that the assistance received 
in response to their questions was adequate. 

In summary, most participants from facilitated schools perceived that they were capable of 
conducting the study independently, and participants from independent schools had few questions while 
they were conducting the study independently. For what tasks did they feel that facilitator assistance was 
needed, respondents from facilitated schools identified only those tasks for which they had received help. 
Respondents from independent schools did not name any tasks for which they would have appreciated 
assistance. Thus, those participants who received less assistance were less likely to name tasks for which 
they thought assistance was required. 



Actual need for assistance. In addition to reviewing the extent to which schools perceived that 
they needed assistance, it is useful to examine other results for more precisely determining their needs in 
completing the School Study. Four variables were taken into consideration for assessing schools’ actual 
need for facilitation: (1) errors or counterintended approaches to carrying out some of the tasks in the 
study, (2) the extent to which Steering Committee chairs at ISs served in a quasi-facilitator capacity, (3) 
the number of schools that switched to facilitated or independent status from their original designated 
status by virtue of the amount of help they required, and (4) the difference in how far FSs and ISs 
progressed in the study. 

(1) While the general perception of both facilitated and independent schools that they could have 
accomplished the study without much assistance may be largely accurate, examination revealed some 
errors in how they approached certain tasks — for these, they would have benefited from additional 
guidance. These errors included 



• not communicating the study’s purpose and inviting buy-in at the outset 

• including too many people on the Steering Committee 

• being unprepared for meetings 

• choosing primary interviewers from among school faculty and staff 
(on or off the Steering Committee) 

• interviewing teachers in pairs 

• too readily interviewing teachers via written survey 

• interviewing students individually 

• not probing for fuller responses from student interviewees 

• interviewing students via written survey 

• not identifying an appropriate district administrator interviewee 

• not sharing Steering Committee duties, i.e., report writing, evenly among members 

• not enlisting helpers from outside the Committee to type interviews and the report 

• not requiring that the entire faculty attend the rating meeting 

• forgetting a step in the study, e.g., the rating meeting 
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Most of these counterintended practices deal with the interviewing step. For the most part, the 
errors mentioned here may be attributed to both facilitated schools and independent schools. Two errant 
practices more apparent in FSs were failing to communicate the study’s purpose at the outset and choosing 
primary interviewers from among school faculty and staff. Three that appear to be problems more 
associated with ISs were not sharing responsibilities evenly among Steering Committee participants, not 
interviewing students in groups, and not requiring the whole faculty to be at the rating meeting. 

(2) It has already been noted that at possibly 5 schools,'^ 1 or 2 Steering Committee members 
performed a substantial portion of the work required to complete the study. Of these 5 schools, 3 were ISs. 
This circumstance generated the question of whether the overcommitment of these particular individuals 
was partially due to their lack of external facilitator assistance. A comparison of the tasks they performed 
to those performed by their overcommitted counterparts at the 2 FSs showed that, in both situations, these 
individuals did several similar tasks not in the domain of facilitators, such as copying, report writing, 
typing, and substantial interviewing. This finding would seem to indicate that their substantial investment 
of energy was more a function of how they delegated Committee work than whether they had a facilitator 
or not. 



However, the chairs at these 3 ISs also did a number of tasks that their counterparts at FSs had not 
done, because they did not have a facilitator’s services as their FS counterparts did. These included 
setting up and leading Steering Committee meetings, introducing the School Study process, assuring the 
logical scheduling of tasks, and facilitating the rating meeting. Thus, the large investment of work by 
chairs at ISs was to some extent, due to having to act jn a quasi-facilitator capacity. In addition, it merits 
mention here that the Steering Committee chair at 2 of these ISs had either little or no contact with their 
designated contact person, reinforcing their lone leadership role in carrying out the study. 

(3) Based on reports from Steering Committee chairs and facilitators/contacts and on overall 
assessments of schools’ efficacy, it may be concluded that 1 facilitated school (FSl) could have completed 
the school study as an independent school and that 1 independent school (ISS) evolved into a facilitated 
school. The assistant principal of FSl led the Steering Committee in such a way that members were 
always prepared and required perhaps no intervention; he remarked that the handbook was easy to follow 
and straightforward enough that he would feel comfortable leading other schools through the process. The 
Steering Committee chair at ISS had many initial set-up questions and requested their contact person to 
take the lead in all interviewing. One other IS (IS2) may or may not be likened to a facilitated school 
because the chair — ^who did a substantial amount of work, including all interviewing — ^was a writing 
cluster leader from the district office. So in a sense, she acted as an external facilitator. Overall, there was 
not a noticeable switch of all ISs to FS status, or vice versa. 

(4) Finally, the degree of progress achieved in the study by both facilitated schools and independent 
schools was evaluated as a measure of their need for facilitator assistance. At the rating meeting, all 5 FSs 
and only 3 ISs engaged in extended discussion about priorities. Yet paradoxically, while 3 ISs progressed 



'^This number excludes FS2, at which^evera/ Steering Committee participants worked a substantial number of hours. 
This analysis is intended to highlight those schools in which a minority of Steering Committee members invested substantially 
more time and work than their co-members. 



O 



50 



40 



to the second part of step 4, planning actions, only 1 FS did so. And further, the only 2 schools that took 
action based on their plans before the opening of the next school year were ISs. 



Revision to the School Study handbook. A change will be made to the handbook to help future facilitators 
address potential problem areas and avert schools from the use of errant practices in the study process. Some 
of the final version handbooks will contain a preface of facilitator guidance notes, these to be distributed only 
to facilitators. This change is reflected in Table 13 at the end of the Results. 



Assessment of the Worth and Outcomes of the School Study 

Worth the effort. Both principals and Steering Committee members were asked whether the 
study ’s outcome was worth the effort (question 2 on the principal form and question 6 on the Steering 
Committee form). Of a possible 5 1 respondents, 48 answered and 3 did not because their forms were 
missing the question. Of the 48 respondents, 34 said yes and 6 were unsure pending future outcomes. 
Seven said no and 1 was ambiguous (5 of the 7 who said no came from FS2). Those who thought that the 
study was worth the effort represented a majority at 9 of the 10 schools (all schools but FS2). 

Outcomes produced or anticipated. Principals and Steering Committee members were asked 
either how the study had benefited/would benefit or had affected/would affect the school (question 3 on 
the principal form and question 7 on the Steering Committee form, respectively). Of a possible 51 
respondents,'* 48 responded and 3 did not because their forms were missing the question. The 48 
respondents gave a total of 66 responses. 

Nineteen of the 66 responses were written in nonspecific terms of raised awareness or expected 
changes to the writing program, i.e., “We have a better understanding of how [the various role groups] 
view our writing process,” and “. . .it will allow us to strengthen our program.” 

Thirty-six responses (from 23 of the 48 respondents) were specific about how awareness was raised 
or particular changes that had occurred or were expected to occur. These responses were categorized as 
specific because they mentioned particular actions and/or the actors involved. Of these 36 responses, the 
most frequent response concerned the need for participation of nonaccountable teachers in writing 
instruction so that writing programs would become fully collaborative and, similarly, the need for 
professional development of all teachers to that end (20 responses). The remaining 16 responses that listed 
desired or expected specific outcomes generated by the study were dispersed among these categories: 
changes to the Consolidated Plan or SBDM Council policy (7 responses), changes in personnel (5 
responses), administrative support and funding for changes (2 responses), and professional development 
(2 responses — ^that is, professional development not mentioned specifically in regards to collaboration as 
above). 

Finally, of the remaining 1 1 of the 66 responses, 7 responses were that there would be no effects 
as a result of the school study (5 of which came from FS2), 3 responses were not sure or no effects yet, 
and 1 response was unclear. 



'*There were actually 52 forms because the principal from FS4 submitted both a principal and Steering Committee 
form. His answers to the question are additive and not contradictory; therefore, they are all counted in this analysis. 
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Summary of the Results 

Ideal practices and counterintended practices in conducting the School Study have been detailed 
throughout the previous discussion of field-test results. From these, best practices have been extrapolated 
and summarized in Table 13. The first section of the table lists guidelines that were already part of the 
handbook, but need to be reemphasized based on how important they proved to be or how weakly they 
were adhered to by field-test schools. The second section lists new and revised guidelines to be 
incorporated in the School Study handbook based on the field-test experience and, again, relative 
importance. 



Table 13: Summary of Lessons Learned from the Field Test 
of the School Study of Writing Instruction 

Pa rt I. Existing Guidelines to Reemphasize in the School Study of Writing Instruction 

Setup 

Emphasize the following recommendations: 

1 . Gamer buy-in from faculty and inform Steering Committee members of the study’s purpose prior to its 
commencement. 

2. Ideally, the Steering Committee should include no more than 5 members. 



Interviews (step 1) 



Emphasize the following recommendations: 

it 3^ It is preferable to solicit s pair ofinterviewers rather than only one. 

4. Interviewers should be parents or individuals not involved with regular operations of the school and 
relatively unknown to interviewees. Avoid committing faculty or other regular staff, whether on the 
Steering Committee or not, to this task. Higher grade level students from another school are an option, but 
not an ideal choice. 

5. Interviewers, particularly those unfamiliar with the writing program, need to be trained. 

6. You may interview more teachers and students than called for in the selection charts if you wish, but keep 
in mind that doing so will require you to spend more time assembling data into the report. 

7. Identify an appropriate district administrator interviewee. Only if that person is adamantly opposed to 
being interviewed should you ask that person to complete the interview in writing. 

8. Random selection of student interviewees is important and necessap^. 

9. Interviewing students in groups of 2-3 is much preferable to interviewing them individually. DO NOT 
interview students via written survey. 

10. Probe student interviewees to elaborate on their answers. Rephrase questions if they do not understand 
what you are asking. 

1 1 . Compile teacher and student interviews. 



Report (step 2) 

Retain the following feature: 

12. A comprehensive sample report (a new one) — ^but exclude the brief sample report. 



Ancillary tasks 

Emphasize the following recommendation: 

13. Enlist clerical helpers fo r typing interview compilations and the report and for large copying jobs 

(continued on the next page) 
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Table 13: Summary of Lessons Learned from the Field Test 
of the School Study of Writing Instruction (continued) 

Part II. Additional and Revised Guidelines to Include in the 
School Study of Writing Instruction 

Setup 

Include the following features: 

1 . A revised time estimate for completing the School Study, i.e., 5-8 weeks 

2. A section of facilitator guidance notes, which would include tips on helping schools to handle such issues as faculty 
buy-in, diplomacy in report writing, and uncooperativeness of district administrators 

3. A section of guidelines/agendas for 2-4 Steering Committee planning meetings 

Include the following recommendations: 

4. Begin by early February to avoid conflicts with events occurring in late spring. 

5. Consider including the feeder school or next level school in your study. 

6. Refrain from having one Steering Committee member do a substantially greater amount of work than the others. Th< 
chair may have to do a little more work than others but not major steps unaided. 

Interviews (step 1) 

Delete the following option and include a recommendation against it: 

7. The option for teachers to complete the interview via written survey 

Include the following feature: 

8. Tips on gaining an interview with reluctant administrators 
Report (step 2) 

Include the following option and features: 

9. The option to write the report in a group meeting of the Steering Committee 

10. A “Table of report sections and data” that links interview questions to report topics for easier writing 

11. A revised time estimate for report writing, i.e., 3-5 hours per contributor 

Rating (step 3) 

Include the following recommendations or options: 

12. Invite faculty to the rating meeting on an expectant rather than a voluntary basis. 

13. If you wish, provide an entire rating guide to each meeting participant. 

Make the following revision: 

14. Identify indicators missing from the rating tools and correct the numbering so that they are synchronous to the 
report. 

Priorities and planning (step 4) 

Redesign this step: 

15. Combine the first part of step 4 into step 3, i.e., transform the faculty rating meeting into both a rating and setting 
priorities meeting, at which the school profile is used as a discussion piece. Make the second part of step 4, planning, 
a supplemental section in the handbook. Thus “step 4” will no longer exist. 

Include the following recommendations: 

16. Progress to planning and action in a timely fashion after the rating/priorities meeting. 

17. If desired, repeat the study annually or as needed to measure writing program progress. 
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CONCLUSIONS 

Overall, positive conclusions are drawn about the self-study process, mainly based on schools’ 
extent of completion, respondents’ estimation of its worth, and anticipated outcomes. Most schools 
completed the study to a satisfactory point and most Steering Committee participants believed that 
conducting the School Study of Writing Instruction (AEL & KDE, 1999) was worth the effort. The time 
commitment required to undertake the study remains an issue of concern. Facilitator assistance was found 
to be highly beneficial. The School Study instrument was shown to be generally valid and replicable, but 
may be improved with various adjustments. 

Value 



The results of the field test indicate that for most schools and individual participants, the School 
Study of Writing Instruction was a valuable and successful experience. Most schools completed the study 
to a satisfactory interim point or to its conclusion. Most progressed to productive faculty discussion and 
preliminary priority work at the rating meeting. Before the close of 1 998-99, about half of the schools had 
already begun step 4, planning actions, and by the summer, 2 schools had acted on one of their new plans. 
Schools that had not begun planning had at least assigned the responsibility to a designated committee or 
the entire faculty for a later time. All schools expressed the intention to complete priority-setting and 
planning before or during the commencement of the 1999-2000 school year. It was promising that 
independent schools were able to conduct their study in about the same amount of time as facilitated 
schools. In sum, most schools demonstrated some degree of progress in decision making about enhancing 
their writing program, thus showing the School Study ’s capacity as a change agent. 

Several concrete outcomes emanated from schools’ self-studies. In response to a feedback form 
question about the effects and benefits of having conducted the School Study, only 7 of 48 principal and 
Steering Committee member respondents anticipated no effects or benefits (5 of these respondents being 
from 1 school, FS2). The 41 who responded otherwise listed many anticipated outcomes, some which 
were specific. The most frequently mentioned effects were (1) awareness of discontinuity in writing 
programs across grades and subject areas and, accordingly, (2) anticipated or already enacted measures to 
reverse that pattern. Other enacted or anticipated outcomes included revisions to school policies, changes 
in personnel, and new professional development. 

Steering Committee participants demonstrated that they were cognizant of a key mechanism of the 
study — ^that of comparing interview responses among role groups — in that the great majority named the 
interviews as the most helpful part of the School Study. Their appreciation for the interviews was further 
demonstrated by the fi-equent practice of including more individuals than the suggested number in the 
interviewee samples. This was surprising because the inclusion of more interviews required more time 
to synthesize the data into the report, already the most time-consuming task of the study. In addition, some 
Steering Committee members found the rating meeting to be a valuable part of the study, demonstrating 
their appreciation for another key dynamic — that of all faculty sharing in writing program awareness. 
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In addition to the conclusion that the School Study is valuable for schools, based upon the evidence 
of productive outcomes as discussed above, it is also established that the School Study was perceivedhy 
Steering Committee participants to be valuable, based on their assessment of its worth relative to the effort 
it required. A sizable majority of principals and Steering Committee members said that the school study 
was worth the effort, representing a majority in 9 of the 10 schools. If Steering Committee members had 
not deemed the School Study to be worthwhile, its impact may have been mitigated during the planning 
step; however, since many participants valued it, it is possible that plaiuied interventions may proceed to 
full implementation. 

Despite these favorable results, it should be acknowledged that the study’s value may be attenuated 
for some schools by the time required to conduct it. Steering Committee members invested a substantial 
amount of time and work over a period of several weeks, among competing commitments and sometimes 
without personal remuneration, to complete their school study. (Yet schools’ delay in starting the study 
compounded the time pressures by forcing the study into an already busy period. Also, few schools 
effectively enlisted the aid of helpers for typing interviews and reports.) That Steering Committee 
members made these time sacrifices and still deemed the study to be worthwhile is a testament to the 
dedication of many teachers, administrators, and parents. 

Another attenuating factor was the method of distributing the $1,500 stipend. The stipend was 
intended to be granted to schools after they had completed the school study and returned their feedback 
forms to the collaborative research team; however, most schools did not understand this and spent time 
securing it at the beginning of the study, a few because they needed it to pay for substitute teachers. The 
possibility of remuneration to future schools to aid them in implementing the study merits consideration. 

FS2's singularity of having a negative experience may be attributed to a combination of several 
possible reasons, hypothesized as follows. First, FS2 seemed to lack the desire to conduct the study at the 
outset, perhaps partially due to deficits in leadership. The Steering Committee chair did not want the 
position, which was essentially forced on her. Also, the principal’s support was not evident. Second, FS2 
was 1 of 2 schools that conducted the school study in the shortest time frame (5 weeks) and the Steering 
Committee members reported that they felt rushed. Third, FS2 had more individuals who spent a 
substantial number of hours (>20 hours) than any of the other schools; 4 of the 1 1 individuals in the field 
test who spent this number of hours were from FS2. The chair was one of these individuals, investing 30- 
40 hours in time-intensive tasks such as interviewing, compiling interviews, and typing the report. Fourth, 
FS2's Steering Committee members were not compensated for their time because they used the stipend 
to pay for substitutes. It is difficult to ascertain whether this circumstance contributed to FS2's low 
evaluation of the School Study, because it is unknown how the majority of schools used their stipend in 
comparison (except that FS3 also used it to hire substitutes and IS5 divided it among Steering Committee 
members). 

In summary, short-term benefits of the School Study have been established, while long-term effects 
have yet to be proven. A later phase in the Kentucky writing project is planned to assess changes in 
writing scores of schools that have undertaken the School Study. At the least, the research team believes 
that because the School Study is based on already existing criteria of the Kentucky writing program, it is 
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an exemplary formative evaluation tool for schools. Even if they cannot demonstrate the degree of 
improvement in writing scores over the next few years that they desire, they will be better equipped to 
ascertain why not and knowledgeably direct improvement efforts. Moreover, they will be able to show 
that they have made attempts at meeting accoimtability standards and perhaps undergird requests for 
additional resources based on systematic data that establish genuine areas of need. Yearly reviews of a 
school’s writing program, as suggested by a few principals in the field test, would further reinforce these 
benefits. 

Facilitator Assistance 

The field test demonstrated that facilitator assistance was advantageous for schools that received 
it. Independent schools conducted the School Study as efficaciously as facilitated schools in some respects 
and less so in other respects. The practices of both types of schools highlighted areas of need for 
facilitation in future schools. 

Based on the consideration of several outcomes, the collaborative research team concluded that 
facilitator assistance was beneficial and preferable to no facilitator assistance: 

• Steering Committees of FSs reported needing and appreciating the facilitator services that they 
received. The services that facilitators rendered alleviated Steering Committee chairs from 
duties associated with spearheading the school study, such as orchestrating and leading 
meetings and introducing the School Study process. The three services that they appreciated 
the most, in descending order, were leading the introductory meeting of the Steering 
Committee, assistance in interviewing, and assistance in report writing. 

• Steering Committee members at some ISs did request facilitative services from their contact 
person. One independent school evolved into a facilitated school by virtue of the amoimt of 
assistance that it requested from its contact person. 

• Some Steering Committee chairpersons at other ISs (who did not request facilitative-level 
services from their contact person) had to assume more responsibilities than their coimterparts 
at facilitated schools presumably because they did not have the services of a facilitator. 
Although Steering Committee chairs at both FSs and ISs performed many of the same tasks, 
it was clear that chairs at ISs also had to perform tasks similar to those performed by the 
facilitators at FSs; with a facilitator, they most likely would have been relieved of these extra 
duties. Essentially, some Steering Committee chairs at ISs became quasi-facilitators. 

• Also, both FSs and ISs engaged in some coimterintended practices for which they could have 
benefited from further guidance, many related to the interviewing step. Three practices that 
were more evident in independent schools than in facilitated schools were (1) the failure to 
divide Steering Committee duties evenly among members, (2) the failure to involve the entire 
faculty in the rating meeting, and (3) the tendency to interview students individually. Yet FSs, 
despite facilitator assistance, experienced difficulties in implementing the School Study as well. 
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These data, rather than mitigate the potential benefits of facilitator assistance, highlight areas 
where assistance can be improved. Facilitators can be trained to redirect some of the errant 
practices identified in the course of this field test. 

In summary, facilitator assistance proved to be valuable in several ways and possesses the potential for 
being so in future schools that undertake the School Study. 

Although facilitator assistance is deemed beneficial and desirable, it is true that schools perceived 
that they did not need considerable assistance. On the whole, both facilitated and independent schools 
initiated contacts with their facilitator or contact person minimally, and most did not request services 
beyond those already offered and provided to them. When asked about the amount of help required, FS 
respondents believed more than IS respondents that they needed facilitator assistance, but two thirds 
nevertheless maintained that they could have accomplished the study unaided. Surprisingly, independent 
schools respondents listed nothing when asked for which tasks they needed more assistance than was 
available. Participants from both types of schools demonstrated (or believed) that they were self- 
efficacious in conducting the School Study to the extent that they were compelled to be so. Still, the ability 
to be self-sufficient does not undermine the merits of facilitator assistance as established earlier; indeed, 
schools should be expected to be somewhat capable of performing a self-study, just as facilitators can be 
expected to enhance the process by which one is done. 

A few paradoxes arose to the conclusion that facilitator assistance is preferable and advantageous. 
Two paradoxes were that independent schools were more likely to initiate their study without prompting 
and a few independent schools progressed further in the study steps than facilitated schools by actuating 
the planning phase. A plausible explanation for these effects is that ISs relied on their own motivation 
because they basically had no one else to rely on. Also, largely self-guided with mainly the handbook at 
their disposal, ISs had no impetus for pausing, thus some proceeded until all steps as outlined in the 
handbook were accomplished. Whereas in FSs, participants relied on the facilitator to prompt and direct 
them. The facilitator’s presence through the rating meeting step may have fostered the implicit assumption 
that the rating meeting was a natural breakpoint or concluding activity, temporarily drawing attention away 
from succeeding steps. However, evidence of greater efficiency by some ISs to begin and conclude the 
study need not negate the several advantages of having a facilitator’s aid at various junctures in the course 
of a school study. 

Another paradox was that 1 FS completed the study with an unfavorable impression despite having 
a facilitator’s assistance (and 1 IS did not complete the study despite the contact person’s offers to help 
at a level akin to facilitator assistance). This result may reveal more about capacity and motivation of 
some schools than the usefulness of facilitation. It is conceivable, for instance, that some low-functioning 
schools, already beset with a host of problems that contribute to their low performance, would lack the 
prerequisite motivation to conduct a self-study, when perhaps they could benefit most from one. 
Therefore, such schools may need intensive facilitation — initial capacity-building work and strong 
advocacy of the study’s benefits to raise faculty buy-in to an acceptable level — ^prior to launching the 
School Study. To these schools, the study would need to be presented well enough so that it is accepted 
at least as a necessity if not as a desirable. Then, the intensive facilitator might provide more than the 
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usual amount of help for the time-intensive tasks of interviewing and report writing. Also, this type of 
facilitator could be instrumental in ensuring that the work is distributed advantageously among Steering 
Committee members as well as typists and interviewers not on the Committee. In a sense, intensive 
facilitation would be a way of infusing a school with temporary capacity so that the School Study could 
be a benefit rather than a burden. 

Validity and Reliability 

The reliability of schools’ reports as judged from the examination was inconclusive — some reports 
were representative of interview data and some were not. Only 2 of the 7 reviewed would be considered 
model reports (and perhaps a third report as well, however a rigorous examination could not be performed 
on it). For the most part, reports were abbreviated and devoid of consistent discriminating detail. 

The deficiency in detail may be attributed to several factors. (1) It seems that report writers 
sometimes considered only one interview data item for report topics as if there were a 1 : 1 relationship 
between them, while ignoring other relevant data items. Indeed, responses to report topics that required 
consideration of fewer data items were more likely to be consonant with the reviewer s responses, whereas 
responses to more abstract or comprehensive topics were less likely to be consonant. Other hypothesized 
and stated reasons for concise and undetailed reports include (2) limited time, (3) assumption of a certain 
level of prior knowledge by report readers (thereby obviating the writers from recording an undue amount 
of detail), (4) reluctance to write so pointedly that a particular person(s) or group is singled out for 
criticism (e.g., writers’ unwillingness to state in report topic 9C that not all faculty required writing 
portfolios), and (5) reluctance to repeat data recorded earlier in the report.'^ 

That brevity and generality seem to have been the prevailing mode for reports necessitates inquiry 
about the utility of the report, i.e., is it reasonable to expect schools to produce model reports, and, do the 
outcomes of a school study depend that much on the representativeness of the report? First, given such 
conditions as limited time, writers’ reluctance to criticize others, and their assumption of readers’ prior 
contextual knowledge of the school, it is probably unreasonable to expect future schools to write 
comprehensive reports. More realistically, report writing will continue to be regarded as one of several 
tasks to be accomplished as efficiently as possible to enable progress to next steps in the study. Second, 
if this shall be the case, whether brevity affects school study outcomes depends on other aspects of the 
study. The report is the springboard to rating and, ultimately, faculty discussion and outcomes. If 
abbreviated summary statements are adequate for stimulating productive discussion and outcomes, then 
brevity in the report should not be a major concern; some precision in data transposition may be 
responsibly sacrificed. Indeed, facilitators or Steering Committee chairs at most schools indicated that 
faculties arrived at meaningful discussion despite perceptions of an unrepresentative report (or interviews). 
Also, most Steering Committee members asserted that the resultant school profile was reflective of their 
writing program, and they anticipated productive changes to the program as a consequence. 



'^The 5 report topics chosen for review occurred midway in the report. Reports from 2 schools showed evidence of 
a history effect, i.e., writers had included much more detail under report topics occurring earlier in the report than in related 
topics occurring later, perhaps to avoid uimecessarily repeating data. 
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On the other hand, a report without detail will deprive non-Steering Committee faculty of the 
essence of the interviews, the product that Steering Committee participants overwhelmingly identified as 
the most helpful aspect of the study — because they powerfully illustrated role groups’ perceptions of the 
writing program. Hopefully, the reinsertion in the final handbook of a “Table of report sections and data” 
that links interview questions to report topics will encourage writers to be more precise about how they 
incorporate the interview data into their reports. 

Validity and reliability should be entertained in the broader context of the School Study of Writing 
Instruction as an overall process, essentially a reconsideration of the study’s value. The research team has 
posed two questions to that end. One, has the independent variable (the School Study) caused an effect 
on the dependent variable (the writing program)? Two, can other schools also conduct the School Study 
of Writing Instruction with a similar measure of success? 

The team’s response to the first question is that immediate effects have been observed — Steering 
Committees found the process to be informative and outcomes were implemented or anticipated. Yet it 
is too early to say what the ultimate effect will be. Year 5 of the Kentucky writing project will entail 
assessing the writing scores of schools that have completed the School Study of Writing Instruction and 
comparing them to scores from earlier years. The ultimate effect may depend on which priorities schools 
choose to act on, since some corrective actions may have more impact than others. 

The team’s response to the second question is that other schools will be able to successfully 
conduct a School Study and find it valuable. Most field-test schools efficaciously conducted the study as 
outlined, at least to a logical interim point of completion, and most participants felt that the process was 
worthwhile. Another supporting argument for favorable replication is that field-test schools customized 
the School Study process in several ways — such as by writing the report in a group forum, interviewing 
more students or teachers than the minimum, involving a district writing leader on the Steering Committee, 
rating as a whole group if warranted by faculty size, beginning priority work during the rating meeting, and 
assigning planning to more than one entity. As well as contributing to improvements in the School Study 
handbook, these adaptations demonstrated the School Study ’s malleability to individual school needs, 
evoking Berman and McLaughlin’s ( 1 975) concept of mutual adaptation (Hord, 1 987). Finally, the School 
Study is a targeted study, designed around 36 indicators of successful writing programs that were identified 
and replicated in many Kentucky schools. Thus, it provides an incentive for schools to evaluate their 
program by an established set of criteria. Yet at the same time, because most of the indicators are not 
exclusive to Kentucky’s writing program, the study is equally applicable to schools across the nation. 
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RECOMMENDATIONS 

The following recommendations are based on the conclusions drawn in this report. They are 
intended to guide implementation decisions of the Kentucky Department of Education and AEL, Inc. 

Recommendation 1. Based on favorable outcomes in most field-test schools, the School Study 
of Writing Instruction is recommended for ongoing implementation in Kentucky schools. Performing this 
study promotes the practices of faculty collaboration in school instruction and of making informed 
decisions based on systematically collected data. Its use also reinforces the importance of including the 
voices of all school stakeholders, especially students, in school change efforts. 

Recommendation 2. Given the time required to conduct the School Study of Writing Instruction, 
the Kentucky Department of Education (KDE) and local administrators should consider supporting schools 
that undertake the study with stipends and/or release time. Also, schools should be urged to begin the 
School Study no later than the first of February to avoid competing demands on faculties’ time. 

Recommendation 3. Facilitator assistance is beneficial and advantageous for maximizing schools’ 
use of the School Study. It is recommended that external facilitators be identified, trained, and made 
available to schools in the fall of 1999. The availability of facilitators will be an opportunity for 
introducing the School Study to schools in addition to being an enhancement to its implementation. 

Recommendation 4. Although the School Study is an optional intervention, KDE should consider 
taking steps to identify and encourage low-performing and continuously declining schools to undertake 
it, which might include providing intensive facilitation beyond the usual level. 

Recommendation 5. The collaborative research team should monitor the progress of future 
schools that undertake the School Study. The team should continue with plans to assess the changes in 
writing scores of participating schools in Year 5 of the Kentucky v^iting project. 

Recommendation 6. A reconfigured “Table of report sections and data” should be included in the 
final School Study of Writing Instruction handbook to increase the representativeness of schools’ reports. 



Recommendation 7. All other revisions to the School Study handbook as listed in Table 1 3 should 
be made to enhance its overall usability and representativeness. These revisions include a modified time 
estimate for conducting the School Study, modified time estimates for certain study steps; changes to the 
orchestration and procedure of certain study steps, mainly the interviews; redesigns to some of the steps; 
and alignment of indicators between various forms. 

Recommendation 8. The School Study of Writing Instruction should be developed as a stand- 
alone product, not only for Kentucky writing programs, but for writing programs nationwide. 
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Recommendation 9. The research into indicators of successful writing programs as the basis for 
the School Study of Writing Instruction may be considered a model for development of other curricular 
needs assessment instruments (AEL, 1997a; AEL, 1997b; Coe et ah, 1999a, 1999b). Not all of this 
research base is specific to writing instruction; some parts are generic to teaching, learning, and school 
efficacy. Thus the School Study may be easily adapted as a self-study needs assessment for content areas 
other than writing. 




61 



51 



REFERENCES 

AEL. (1995). AEL technical proposal for operation of the regional educational laboratory in the 
Appalachian region for 1996-2000 (pp. 5.2-1 - 5.2-8). (Submitted in response to RFP 95-040). 
Charleston, WV: Author. 

AEL. (1997a). AEL activities in Kentucky, 1996-97. Unpublished manuscript. 

ART- (1997b). Indicators of successful school writing programs in Kentucky: Executive summary of 
preliminary findings. Charleston, WV : Author. 

AEL. (1998). Crosswalk: Proposal to present practice: Task 2: Project 1: Designing professional 
development for portfolio improvement. Unpublished manuscript. 

AEL, & Kentucky Department of Education. (1999). School study of writing instruction: Field test 
version. [Process manual]. Charleston, WV: AEL. 

Berman, P., & McLaughlin, M. W. (1975). Federal programs supporting educational change: Vol.lV: 
The findings in review. Santa Monica, CA: Rand. 

Burczyk, R. (1998). 1995-96 Locale Types (Johnson Codes): Designation for schools in AEL’s 

region. Charleston, WV : AEL. 

Coe, P., Keyes, M., Lewis, S., Meehan, M., Orletsky, S., Rigney, S., Runge, C., & Whitaker, J. (1999a). 
Development and validation of successful writing program indicators based on research in 
continuously improving and continuously declining schools. Paper presented at the meeting of the 
American Educational Research Association, Montreal, Canada. 

Coe, P., Keyes, M., Meehan, M., Orletsky, S., Lewis, S., Rigney, S., Runge, C., & Whitaker, J. (1999b). 
Development and validation of successful writing program indicators based on research in 
continuously improving and continuously declining schools: Report of the Kentucky state writing 
project. Charleston, WV: AEL. 

Hord, S. (1987). Evaluating educational innovation. New York: CroomHelm. 

Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards: 
How to assess evaluations of educational programs (2"‘‘ ed.). Thousand Oaks, CA: Sage. 

Kentucky Department of Education. (1990). 1990 Kentucky school laws. Frankfort, KY: Author. 

Kentucky Department of Education. ( 1 994a). Kentucky writing portfolio: Writing portfolio development 
teacher ’s handbook (2"‘‘ ed.). Frankfort, KY : Author. 



O 



62 



52 



Kentucky Department of Education. (1994b). Writingportfolio procedures Frankfort, 

KY: Author. 

Kentucky Department of Education. (1996). 1996-97 District assessment coordinator ’s implementation 
guide. Frankfort, KY; Author. 

Kentucky Department of Education (1998a, December). Kentucky school and district accountability 
results, Accountability Cycle 3 (1994-95 to 1997-98). Frankfort, KY: Author. 

Kentucky Department of Education. (1998b). 1998 Kentucky school laws. Frankfort, KY: Author. 

Kentucky Department of Education. (1999). Writing portfolio procedures (703 KAR5:0\0). Frankfort, 
KY: Author. 

Lewis, S., Runge, C., Whitaker, J., Coe, P., Keyes, M., Meehan, M., Orletsky, S., Parrish, P. (1999, 
October). Development and validation of successful writing program indicators based on research 
in continuously improving and continuously declining schools in Kentucky. Paper presented at the 
meeting of the National Rural Education Association, Colorado Springs, CO. 

Parrish, P., Keyes, M., Coe, P., Tolbert, S., Orletsky, S., Lewis, S., Runge, C., Meehan, M., Whitaker, J. 
(1999a). Pilot test of the School Study of Writing Instruction : A self-study needs assessment 
instrument. Charleston, WV : AEL. 

Parrish, P., Tolbert, S., Coe, P., Keyes, M., Lewis, S., Meehan, M., Orletsky, S., Runge, C., Whitaker, J. 
(1999b, November). Pilot test of a writing program self-study instrument for schools. Paper 
presented at the meeting of the American Evaluation Association, Orlando, FL. 

Quality Education Data. (1998). QED’s state school guides: Kentucky (lb'** ed.). Denver, CO: 

Author. 




63 



APPENDIXES 



O 

ERLC 

hiMfiiifftifiiTiaaaa 



APPENDIX A: 



Table of Observed Score Differences, t-Values, p-Values, and Significance Levels of All 
Indicators Between the Continuously Improving and Continuously Declining Schools 
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Table of Observed Score Differences, t-Values, p-Values, and Significance Levels 
of All Indicators Between the Continuously Improving and Continuously Declining Schools 
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Indicators of Successful 
School Writing Programs 
In Kentucky: 

Executive Summary of 
Preliminary Findings 



A joint project of Appalachia Educational Laboratory 
and the Kentucky Department of Education 




For the past two years, a collaborative study 
team of researchers has been visiting schools 
across Kentucky in an attempt to answer this 
question; 

What practices and conditions produce 
consistent improvement in students* port- 
folio writings, as measured by the KIRIS 
portfolio assessment? 

Theyve interviewed more than 100 teachers, 
200 randomly selected students, and 50 admin- 
istrators in schools with various writing success 
rates — some consistently successful in raising 
scores, others consistently unsuccessful. Mem- 
bers of the research team include staff from the 
Appalachia Educational Laboratory (AEL), 
along with staffand consultants from the Ken- 
tucky Writing Program. 

This first phase of the five-year study — a joint 
project of AEL and the Kentucky Department 
of Education — seeks to identify indicators of 
high performance. This report summarizes the 
conditions most frequendy observed in the more 
successful schools. Scores improved consis- 
tently in these schools over a four-year period 
beginning in 1992, although beginning scores 



were not necessarily high. The study examines 
whether schools evolve through a common se- 
quence of instructional practices and conditions 
as they grow increasingly effective in writing in- 
struction. Ultimately, the team plans to use the 
indicators to create a process that schools can 
use to plan improvements in their writing in- 
struction. Department staff also will use the 
information to evaluate and plan future profes- 
sional development offerings. 

The following indicators are those most fre- 
quently observed in schools that improved writ- 
ing portfolio scores over two consecutive ac- 
countability cycles. The research team regards 
these indicators as tentative until confirmed by 
further data collection and analysis. 

^ 

Tentative Indicators Reflecting 
School/Oistrict Support of the 
Writing Program 

• The district demonstrates commitment to 
the writing program by 

- allocating resources to professional devel- 
opment or technical assistance; 

- assigning program oversight to qualified 
personnel and allocating sufficient time for 
effective oversight; 

- compensating cluster leaders through ad- 
ditional pay or released time; and 

- in some districts, establishing policies re- 
quiring portfolio completion for promo- 
tion or graduation. 

• The principal actively supports writing in- 
struction by providing resources, technical 
assistance, and/or professional development, 
and by providing substitutes to allow teach- 
ers to meet together to score portfolios. 

• A high degree of collegiality is evident among 
teachers. Portfolio scoring is a team event 
featuring discussion of instructional gaps, 
weaknesses, and strengths evident in student 
writing. 
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• Language arts teachers at the ac* 
countability grades are confident of 
their understanding of writing port- 
folio requirements; all have received 
professional development in the 
writing process, portfolio develop- 
ment, and scoring. They use The 
Writing Por^lio Teacher’s Handbook, 
2nd Edition, and feel they have reli- 
able sources of information and as- 
sistance when needed. 

• School writing leaders are satisfied 
with the level of training and sup- 
port they have received to assist other 
teachers wdth portfolio developmenL 

• Most language ans teachers at the 
non-accountability grade levels par- 
ticipate in professional development 
on writing instruction and portfo- 
lio development. 

^ ^ 

Tentative Indicators 

Reflecting Instructional 
Strategies 

• Students write frequently in all sub- 
jects, and the writing is integrated 
into instruction* 

• Teachers in most grades and content 
areas give writing assignments that 
have the potential of contributing to 
students* “working portfolios.** 

• Teachers promote peer conferencl ng 
as well as student-teacher con- 
ferencing; students feel comfortable 
receiving help from and providing 
help to other students* 

• Teachers spend substantial time on 
prewriting aaivitics. 

• Teachers focus on developing writ- 
ers rather than developing portfolios 
so that, when it is time to put a port- 
folio together, students have a num- 
ber of pieces from which to choose, 
most of which were written as a 
natural outcome of their studies. 

• Teachers provide latitude for stu- 
dents to choose topics and/or for- 
mats when they write* 

O 



• Teachers model parts of the writing 
process as they work with students. 

• The mechanics ofwriting (grammar, 
spelling, punctuation) are taught in 
the context of writing, rather than 
as unrelated drills and worksheets. 
Lessons are crafted to address needs 
evident in student writing. 

Two indicators are less frequently 



observed, but appear to have a power- 
ful impact on students: 

• Students write for “real-world” au- 
diences and for real purposes. These 
audiences read and respond in some 
fashion to the writing. 

• Teachers share their own writing 
with students and invite students* 
critiques. 



Students Talk About Their School Writing Experience 



An important part of the study is 
the student interview. Students were 
interviewed to assess the effect of 
school conditions and practices on 
student attitudes and behavior with 
respect to writing. Students talked 
about their school writing experience 
and their perceptions of themselves as 
writers. In schools with continuously 
improving portfolio scores, student, 
teacher, and administrator accounts 
of writing instruction were highly 
consistent. 

In schools with continuously im- 
proving portfolio scores, students 
commonly 

• speak of themselves as writers, 
rather than students who must 
complete writing assignments; 

• speak of writing as a routine part 
of their school day, rather than as 
separate tasks done to produce a 
portfolio; 

• expect that writing competence 
will be necessary in adult life, what- 
ever career path they may follow; 



describe substantive ways their 
writing has improved from one 
year to the next, including choice 
of topics, organization, use of sup- 
porting details, spelling, grammar, 
and punctuation; 

express confidence that most stu- 
dents — including themselves — 
can become proficient writers with 
sufficient effort; 

work with their peers on a regular 
basis to improve their writing — 
asking questions to clarify the 
authors intent, as well as giving 
and receiving suggestions for im- 
provement; 

carry a folder of written work — 
their “working portfolio” — ^with 
them from year to year, and use it 
either to compare earlier work 
with current writing or to develop 
earlier work for current portfolios; 
and 

are familiar with the Kentucky 
benchmarks for novice through 
distinguished writing and use the 
vocabulary of the writing process. 



To learn more about the preliminary findings in this study,* visit AEL*s web 
site at http://www.ael.org/rel/state/ky/kyrpt97,htm. 



*A collaborative research project begun in 1 996, conducted by staff members of Appalachia 
Educational Laboratory, the Kentucky Department of Education, and the Kentucky Regional 
Writing Consultants. A report of the preliminary findings was written in November 1997. 



This publication is bsued on work sponsored wholly or in pan by the Office of Educational 
Research and Improvement, U. S. Department of Education, under contract number 
RJ96006001. Its contents do not necessarily reflect the views of OERl, the Department, or 
any other agency of the U. S. government. 
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Various Data Sources 

(Principal’s Feedback form, Steering Committee Member Feedback form [t>vo versions], 
Phone Interview Protocol for Steering Committee Chairs) 
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PRINCIPAL’S FEEDBACK 
Kentucky School Study of Writing Instruction 




School name 



Date 



Thank you for helping with the field test of this study. Please take a few minutes to give 
us your reaction to it, so that we may identify areas that need to be improved and how 
widely it should be disseminated. 

************** 



1. Do you feel the writing program profile produced by the study is an accurate picture of 
your writing program’s strengths and weaknesses? If not, what do you think may have 
caused the inaccuracy? 



2. Do you feel the outcome of the study was worth the effort? 



3. How do you expect the study has benefitted or will benefit your school? 



4. What, if any, improvements or other suggestions can you offer to strengthen this study 
process? 



January, 1999 



STEERING COMMITTEE MEMBER FEEDBACK 
Facilitated field test 

Kentucky School Study of Writing Instruction 



Position (ex. Teach 9th grade social studies) 

School name Date 

We appreciate your help in field testing the School Study of Writing Instruction. Now that you have 
completed the study, please write brief responses to the questions below. We will use your experience 
to improve the process before it is prepared for a wider use. 

1. Please estimate the total amount of time you nersonallv spent on matters having to do with this 
study from its beginning to its end. 



2. Please list the the task(s) that required the most time and estimate how much time each of these 
required. 



3. How many steering committee meetings did you hold? What was done at each? 



4. Do you think you could have carried out this study without outside help? What, if any, tasks 
would have been difficult without outside assistance? 
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5. Do you feel the writing program profile produced by the study is an accurate picture of your 
school’s writing program’s strengths and weaknesses? If not, what inaccuracies were there and 
what about the process do you think may have caused them? 



6. Do you feel the outcome of the study was worth the effort required to complete it? 



7. How do you feel the study has affected or will affect the school? 



8. Which parts of the study process do you consider most helpful? Why? 



9. Which parts of the study process do you consider least helpful? Why? 



A 



O 
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OVER 



10. What, if any improvements or suggestions (other than those previously mentioned) can you 
offer to strengthen this study process? 



11. Did you look at the sample report included with the study handbook? If so, was it helpful? 
Would you recommend that it continue to be included with the handbook? 



January, 1999 
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STEERING COMMITTEE MEMBER FEEDBACK 
Independent field test 

Kentucky School Study of Writing Instruction 

Your position (ex. 5th grade l.a. teacher) 

School name Date 

We appreciate your help in field testing the School Study of Writing Instruction. Now that you 
have completed the study, please write brief responses to the questions below. We will use your 
experience to improve the process before it is prepared for a wider use. 



1. Please estimate the total amount of time vou nersonallv spent on matters having to do with this 
study from its beginning to its end. 



2. List the the task(s) that required the most time and estimate how much time each of these 
required. 



3. How many steering committee meetings did you hold? What was done at each? 



4. At which points (if any) in the study did you need to contact someone with questions? Which 
tasks (if any) did you feel the need for more assistance than was available? 



OVER 



5. Do you feel the wi iting program profile produced by the study is an accurate picture of your 
school’s writing program’s strengths and weaknesses? If not, what inaccuracies were there and 
what about the process do you think may have caused them? 



6. Do you feel the outcome of the study was worth the effort required to complete it? 



7. How do you feel the study has affected or will affect the school? 



8. Which parts of the study process do you consider most helpful? Why? 



9. Which parts of the study process do you consider least helpful? Why? 



10. What, if any, improvements or suggestions (other than those previously mentioned) can you 
offer to strengthen this study process? 



11. Did you look at the sample report included with the study handbook? If so, was it helpful? 
Would you recommend that it continue to be included with the handbook? 



January, 1999 
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Phone Interview Protocol for Steering Committee Chairs 
about the Field Test of the School Study . . . 

Planning 

1 . How many planning meetings did the Steering Committee hold? 

2. Who presented/explained the manual to the Steering Committee? How did she/you go about 
it? 

3. What types of things did the Steering Committee do in their planning meetings? For 

instance, did they preassign faculty small groups in preparation for the faculty rating session 
(step 3)? . 

Interviews /step 11 

4. Was choosing the interviewer(s) easy or hard? How many interviewers did the Steering 
Committee choose? 

5. Was the Committee able to employ random selection of teacher interviewees as described in 
the manual, or did they have to “modify the randomness” of the selection process in order to 
ensure a cross section of different types of teachers? 

6. Were students interviewed individually or in groups? How well did it work? 

7. Were the interviewers able to stay within about a class period per interview? How long did 
it take them altogether to do the interviews? How long did it take to type the interviews 
(and compile the teacher and student interviews onto one form each, if this was done)? 

Report (step 21 

8. How long did it take each Steering Committee member to complete her/his section of the 
report? How long did the typing and compiling of the complete report take? 

Rating Session (step 31 

9. Describe the rating session. How long did it take? How far did the faculty get - did they 

get to the school profile, to preliminary work on setting priorities perhaps ? 

Priorities and Actions /step 4') 

10. (Priorities, part 1 of step 4): Did the faculty-at-large or separate committee set priorities? 

Other 

1 1 . How many were on the Steering Committee? What is the estimated total time per individual 
Committee member spent during this process, including time spent in individual reading of 
the manual, preparation for meetings, etc.? What step/activity seemed to take the most 
time? 
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Phone Interview Protocol for Steering Committee Chairs 
about the Field Test of the School Study . . p. 2 

12. What was the time frame for the total process, e.g., four weeks? five weeks?, from the time 
that the Steering Committee was formed until the “end” of the study (The end may be 
defined as the faculty rating session, setting priorities, or planning actions, depending on 
how far the school progressed.)? 

13. Do you have any questions for me? 
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Time and Task Summary 


What is the task? 


How long does it take? 


Who docs it? 


Select people to be interviewed, 
schedule interviews, select and, if 
necessary, train interviewers 


? Time depends on ease of 
recruiting interviewers and on 
whether training is necessary. 


Steering committee 


Complete an interview about 
writing instruction 


About one hour for each interview 


Four to 8 teachers and their 
interviewers (persons from outside 
the school) 


Interview a random sample of 8-12 
students (total) from the 
accountable grade, previous and 
following grades. 


About four hours 
Students can be interviewed in 
groups of two or three. Each 
interview takes 30 to 45 minutes. 


Students and interviewer(s) (person 
or persons from outside the 
school.) 


Gather data about resources 
available to the school’s writing 
program 


About one hour 


j* 

Principal or his/her designee 


Interview principal and district 
administrator 


One hour each 


Steering committee member(s) 


Develop a summary report, using 
survey results, resource inventory 
and 

student interviews. 


One to two hours per person 


Steering committee members and 
additional people, if needed 


Make copies of the report for all 
faculty members. 


One to two hours. 


Support staff persons. 


Prepare to lead faculty meeting to 
assess writing program 


One hour 


Steering committee 


Using the report, assess school’s 
strengths and needs on the analysis 
form provided. 


Three hours 


Entire faculty led by the steering 
committee**^ 


(Optional) 

Prepare to facilitate faculty meeting 
to choose improvement priorities. 
Copy materials needed in meeting. 


One hour 


Steering committee 


(Optional) 

Decide on improvement priorities. 


Three hours 


Steering committee and faculty 



'The faculty will work in small groups on different sections of the analysis form. This approach deepens the 
faculty’s understanding of their writing program’s strengths and needs and increases commitment to improvements. 

^Schools that are so large that it is unworkable to include all faculty in this meeting can select a representative 
sample of teachers from each department or grade. A group of 18 to 24 people is adequate to complete the rating 
task. However, try to include as many teachers as possible in order to build understanding of the writing program 
across the faculty. 
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School Study of Writing Instruction 
Report Form 



NAME OF SCHOOL: 

REPORT COMPLETED: (Month) (Year) 

NAMES OF PEOPLE WHO DEVELOPED THIS REPORT: 



See the guide for completing this report before beginning work. 
When completed, make copies for all faculty members and distribute. 



O 
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Indicators of Writing Portfolio Improvement 



1. Administrative support 
a. District level 



b. School level 



Other (information other than that obtained through the study): 
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2. Availability and effective use of quality training/technical assistance 



A. Training of writing leaders 



- Building level 



- District level 



Other: 



B. Training of portfolio-accountable teachers (language arts teachers in departmentalized 
schools; otherwise, all teachers responsible for teaching writing at the accountable grade 
level) 



Other: 
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C. Extent to which ^ teachers are trained in writing instruction 
a. Administrator perspective: 



b. Teacher perspective: 



Other: 



D. Attention to writing that is appropriate for content areas other than language arts: 
a. Administrator perspective: 



b. Teacher perspective: 



Other: 



O 
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E. Teacher training in scoring 

a. Administrator perspective: 



b. Teacher perspective: 



Other: 



F. Portfolios scored by a team of teachers with diverse membership (more than just 
accountable grade or English teachers) 

a. Administrator perspective: 



b. Teacher perspective: 



Other: 
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G. Evidence of mentoring/informal professional development 
a. Administrator perspective: 



b. Teacher perspective: 



Other: 



H. Level of strategic collaboration in professional development 
a. Administrator perspective: 



b. Teacher perspective: 











Other: 
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3. Coordination 

A. Coordination across grade levels including between elementary and middle school; 
middle and high school 

a. Administrator perspective: 



b. Teacher perspective: 



Other: 



B. Coordination across subject areas 
a. Administrator perspective: 



b. Teacher perspective: 



Other: 
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4. Su pportive school climate and easy, frequent communication among staff 
a. Administrator perspective: 



b. Teacher perspective: 



Other: 



O 
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5. Evidence of communication with families about writing instruction and family support of 
the writing program 

a. Administrator perspective: 



b. Teacher perspective: 



c. Student perspective 



Other: 



6. Evidence of use of community resources in the writing program 
a. Administrator perspective: 



b. Teacher perspective: 



O 
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c. Student perspective: 



Other: 
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7. Focus and intensity of writing instruction programs and degree to which writing and/or 
writing portfolios are a clear school priority 



a. Administrator perspective: 



b. Teacher perspective 



Other: 
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Instructional strategies 

8. Use of The Writing Portfolio Teacher ’s Handbook. 2nd Edition as a resource by all teachers 
teaching writing 

Teacher perspective: 



Other: 



9. Evidence that teachers and students focus on writing 

A. Evidence that portfolios are not the sole focus of the writing program 
a. Teacher perspective: 



b. Student perspective: 



I 

■ Other: 
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B. Evidence that students speak of themselves as writers rather than simply completers of 
writing assignments 

Student perspective only: 



Other: 



C. Evidence that students have the opportunity to compare current writing with writing 
completed earlier — even in previous years — and can describe how their writing has 
improved 

a. Student perspective; 



b. Teacher perspective: 



Other: 
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D. Evidence that teachers require more challenging work from students than they once did 
because students come better prepared 

a. Student perspective: 



b. Teacher perspective: 



Other: 



E. Evidence of students’ confidence that they can eventually reach proficiency 
Student perspective only: 



Other: 
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10. Evidence that students routinely use the writing process and know the steps in the process. 
A. Students refer to the steps in the writing process when talking about their own writing. 
Student perspective only: 



B. Students are aware of criteria forjudging portfolios (and ways in which they learned 
these criteria) 

a. Student perspective: 



b. Teacher perspective: 



Other: 



ioo 

o 
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C. Use of computer to facilitate writing 
a. Student perspective: 



b. Teacher perspective: 



Other: 



D. Use of feedback to improve writing (conferencing with teachers, family, or other 
students) 

a. Student perspective: 



b. Teacher perspective: 



Other: 




\ ' 
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11. Teacher writing 

A. Evidence that teachers write and share their writing with students (writing assignments, 
journal writing, etc.) 

a. Teacher perspective: 



b. Student perspective: 



Other: 



B. Evidence that teachers introduce students to the many functions of writing in adult life 
a. Teacher perspective: 



b. Student perspective: 



Other; 
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12. Evidence of student choice of topics and formats for particular writing assignments 
a. Teacher perspective: 



b. Student perspective: 



Other: 



IS'j Evidence of real- world writing 

A. Evidence of students having the opportunity to use writing for real-world purposes 
a. Teacher perspective: 



b. Student perspective: 



Other: 
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B. Evidence that students are aware of the audiences they are writing for and that the audiences 
are important to them 

a. Student perspective: 



b. Teacher perspective: 



Other: 



14. Evidence that writing instruction is integrated into general instruction 

A. Teachers assign reading as a source of ideas and models for \\Titing for a variety of 
purposes 

a. Teacher perspective: 



b. Student perspective: 



Other: 



O 
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B. Mechanics taught in context of writing 
a. Teacher perspective: 



b. Student perspective: 



Other: 
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15. Evidence of consistency of the messages students receive about writing from different 
teachers 



a. Student perspective: 



b. Teacher perspective: 



Other: 



1 6. Consistency among student/teacher/administrator perspectives (overall) 



O 
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SCHOOL STUDY OF WRITING INSTRUCTION 
RATING GUIDE 



1 



I Administrative support 

A. District level — Administrative support includes any formal or informal evidence 
that the district administration gives the writing program high priority in the district curriculum. 
Such support may look different from district to district depending on the resources of the 
district, the number of central office staff, and local history and culture. The key criterion is that, 
for this district, consistent effort is going into proactive encouragement of the writing program. 

High — The district supports the program by seeking district policies intended to stimulate 
writing improvement, encouraging and supporting school initiatives to improve writing 
(e.g., providing funds for professional development above that expected by the state, 
providing release time for mentoring), recruiting and/or training expert, facilitative 
central office staff to provide technical assistance and mentor school staff, organizing 
portfolio scoring for the district to ensure accuracy, organizing cross-school or cross- 
grade conversations about writing instruction. 

Medium — The district supports the program by providing assistance to schools in 
support of the writing program that is perceived by the school staff as more facilitative 
than is provided in other curriculum areas (for instance, encouraging schools to provide 
more training for teachers than is generally expected but requiring them to use school 
professional development funds for the purpose), identifying some of the more successful 
practitioners to facilitate the writing program at the district level. 

Low — The district does only the minimum required by the state, assigning portfolio 
oversight to personnel who lack clear qualifications or successful practice, maintaining 
bureaucratic oversight of schools' writing programs with little or no technical assistance. 

Example of a medium to low district we have visited: The district had a cluster 
leader who served as mentor to the writing leader in the school but subsequently 
assigned that staff member to a different assignment that prevents her from 
visiting schools and mentoring writing leaders. The district encourages schools to 
use their professional development funds to send the writing leader and intern 
teachers to intensive professional development workshops not available to most 
teachers but does not make any district funds available for the purpose. No 
stipends are provided, but expenses are paid — in sharp distinction to what is 
provided for “extra” professional development in other areas. 

B. School level — Administrative support includes any formal or informal evidence that the 

school administration gives the writing program high priority in the school curriculum. 
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High — School administrators provide active leadership; changing schedules to allow 
additional time for planning, conferencing or dialogue among teachers or time for 
instructional aides or volunteers to conference with students; providing extra time and/or 
payment for lead teachers, encouraging generous allocations for professional 
development and instructional resources in writing in the school budget and finding 
additional funds when needed to take advantage of unanticipated opportunities; 
participating in professional development with teachers; serving as readers for student 
writing, participating in portfolio scoring, initiating efforts to bring student work to real 
world audiences. 

Example from a school we visited: To illustrate that active principal leadership 
need not always be hands-on, we visited a school where the principal did not 
personally participate in the writing program but encouraged teachers at all grade 
levels to take as much training as possible in the writing process and in portfolio 
development. In addition, he persuaded the school council to allocate a full-time 
aide to the fourth grade, to allow extra planning time for fourth-grade teachers and 
an additional adult to conference with students about their writing. He delegated 
full responsibility for the writing program to the fourth-grade teachers, who were 
regarded as the school's writing leaders, but he made sure they had more resources 
than teachers at other grade levels (for instance, were able to take more field trips 
if they felt it was necessary for the writing program) and celebrated the creative 
ideas they had to stimulate student writing and to share that writing with real- 
world audiences for purposes that were important to the students. 

Medium — School administrators make sure that teachers are provided with up-to-date 
information, professional development and hands-on technical assistance from competent 
district staff; provide release time or stipends for extra duties associated with the writing 
program, such as portfolio scoring; allow and encourage teachers to participate in 
professional development over and above that called for in the school’s P.D. Plan; but let 
the central office take the lead in encouraging the writing program and do not change the 
school’s staffing or organization to improve the writing program. 

Example: We visited a high school where the principal had no direct involvement 
in the writing program but rather delegated responsibility for encouraging the 
program to an assistant principal and the cluster leader. The cluster leader took 
the initiative to approach the KDE Writing Program to answer questions she had 
about whether particular types of real-world writing were appropriate to include in 
the portfolio, and she was able to involve non-language arts teachers in the writing 
program to an unusual degree, but primarily on the basis of her own reputation in 
the school, not administrative support. 

Low — School administrators obey state directives at a minimum level, perhaps 
monitoring implementation of the writing program without actively assisting in it, failing 
to encourage professional development in writing for most teachers and providing no 
special resources to enhance writing instruction or portfolio development. 
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2. Availability and use of quality professional developmcnt/technical assistance 

This indicator refers to the entire writing program, not just portfolio development 
and/or scoring. Consider both formal and informal opportunities. Formal professional 
development denotes planned experiences led by one or more experts in the field. Informal 
professional development denotes mentoring relationships, coaching, electronic networks, peer 
dialogues about writing instruction sustained over time, technical assistance provided on request 
to individual teachers. The first three items below ask for a global assessment of the quality and 
quantity of professional development (in the broad sense) received by teachers at the school. The 
remaining items under this heading require judgments about specific types of training or informal 
professional development opportunities of particular interest to this project. 

A. For writing leaders (cluster leaders or other teachers assigned responsibility for the 
writing program) 

—BUILDING LEVEL 



High — Writing leaders are encouraged to participate in much more than the minimum 
professional development on writing expected of those in their positions; they seek out 
appropriate conferences and workshops to attend, have a support network (within the district, 
with peers in other districts, with consultants, and with KDE Writing Program staff, and have 
sources of on-demand technical assistance and/or coaching, which they find helpful. In turn 
they provide more training and support than is expected of them to the teachers with whom 
they work and encourage them to seek additional professional development in writing 
(strengthening their own skills by instructing others). 

Medium — Writing leaders are encouraged to seek out appropriate conferences, workshops, 
and technical assistance but are unable to avail themselves of all appropriate opportunities 
because of school or district limitations (e.g., financial limits, isolation, lack of substitutes, 
etc.) or personal limitations (e.g., a demanding family situation). Opportunities to share what 
they have learned with teachers in the school may also be limited, for some of the same 
reasons. They have a limited support network to call on. 



Example: One teacher in the school is the designated writing program leader and is 
encouraged to attend conferences and workshops but has limiti 



A . 

LWVa ivilvy 






IS 



available. She has never attended the summer workshop at Western Kentucky University, 
for instance, even though the district is less than 200 miles from Bowling Green. One or 
two people in or near the district constitute her support network and sources of 
information; not being a cluster leader, she does not have access to the networking 
resources provided through the state department of education. She has structured the 
school’s writing program around writing instruction in her own field (Title I), and she 
clearly understands the writing process better than any of the teachers with whom she 
works. 
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Low — Writing leaders have attended the minimum amount of professional development on 
writing expected of people with their responsibilities. They have either not sought or have 
not had the support necessary to participate in additional training. They have formed no 
support networks. They have provided minimum support to other teachers in the school. 



B. For portfolio accountable teachers (teachers responsible for teaching writing at the 
accountable grade; in departmentalized schools, teachers of language arts) 

High — Teachers responsible for writing at the accountable grade level have participated in 
much more than the minimum professional development on writing expected of those in their 
positions (including training in portfolio scoring). They attend conferences, have a support 
network and are able to use on-demand technical assistance and/or coaching, which has been 
helpful. They feel some responsibility to let other teachers who are less accountable for 
student portfolios know how to work with students to improve portfolios. 

Medium — Only some teachers of writing at the accountable grade level have received 
extensive training; others have had professional development in the writing process and in 
portfolio scoring but tend to rely on a few leaders, who have developed a support network to 
rely on and who are willing to support the less involved teachers. 

Low — Writing teachers have attended the minimum amount of professional development on 
writing expected of people with their responsibilities (if they are cluster leaders) or no more 
professional development than most other Kentucky writing teachers (if they are not cluster 
leaders). They have either not sought or have not had the support necessary to participate in 
additional training. They have formed no support networks. They have provided minimum 
support to other teachers in the school. 



C. For other teachers (Other teachers include both teachers in subject areas other than 
language arts and language arts teachers in ndn-accountable grades.) 

High — Interviewees reported that teachers have participated in much more than the minimum 
professional development on writing expected of those in their positions (often because the 
school or district has sponsored extensive inservice professional development in the writing 
process). They attend conferences, support one another routinely, and have resources for on- 
demand technical assistance and/or coaching which they find helpful. 

Example of teachers supporting one another routinely: The primary teachers at one 
elementary school we visited have worked together to design and develop a cumulative 
portfolio, starting with only a few pieces in the first year of the primary program and 
adding additional pieces each year so that students leaving the primary program and 
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entering fourth grade have a full working portfolio. The fourth-grade teachers work closely 
with the primary teachers to help them develop these working portfolios, and students report 
that putting together their final primary portfolio was a more rigorous learning experience 
than putting together the final, fourth-grade portfolio. 

Medium — Most teachers interviewed have participated in workshops or other professional 
development on the writing process and on scoring portfolios, but content area teachers or 
teachers at other than the accountable grade level make little use of the training, as they 
depend on writing teachers at the accountable grade level to take primary responsibility for 
the writing program and for portfolios. There is minimal networking, as writing teachers talk 
with other teachers about what they can do to strengthen students’ writing skills and their 
portfolios. 

Example: At most of the departmentalized schools we have visited writing in the content 
areas has been the most problematic part of the writing program even when the content 
area teachers were convinced that writing was important to their disciplines, which they 
frequently were not. Finding appropriate professional development opportunities has 
often been difficult. In a high school where the great majority of the content area teachers 
cooperated willingly with the English department in writing instruction, the English 
teachers themselves asked that the content area teachers follow the writing process only 
through the first draft stage. After that, if the piece was considered appropriate for a 
portfolio, the piece would be sent to the English department and English teachers would 
work with the student to edit and publish the piece for inclusion in the portfolio. At other 
high schools, language arts teachers complained that content area teachers would send 
them writing pieces that simply were not good enough to be included in a portfolio or that 
were based on inadequate prompts. 

Low — Teachers have attended the minimum amount of professional development on writing 
expected of teachers in their content area or grade. They have either not sought or have not 
had the support necessary to participate in additional training. They have formed no support 
networks. They have neither sought nor provided support to other teachers in the school. 



D. Attention to writing that is appropriate for content areas other than language 
arts 

High — Recent professional development opportunities for all teachers have featured ways to 
incorporate writing authentically into content areas other than language arts, and the school 
writing leader or leaders have followed up with continued technical assistance to help 
teachers use the instruction. In addition, teachers in content areas other than language arts 
may have been encouraged to take intensive instruction in writing for their content areas and 
have subsequently shared what they have learned with others in the school. 

Medium — Some but not all teachers have had the opportunity for professional development 
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emphasizing how to incorporate writing into content areas other than language arts. These 
opportunities may have been made available only to non-language arts teachers who have 
expressed a particular interest in using writing to enhance their instruction. 

Low — There is no evidence that attention to writing that is appropriate for content areas other 
than language arts has been identified as a school need, and no one appears to be seeking 
professional development in this area. 

E. Teacher training and participation in portfolio scoring 

High — Portfolio scoring training is available to all teachers in the school but is voluntary for 
teachers who are not on the scoring team. The portfolio team includes teachers from subject 
matters other than language arts and/or from non-accountable grade levels, and the team’s 
membership changes fairly frequently, to encourage participation by any teacher who is 
interested. 

Medium — Only teachers who are on the scoring team receive training in portfolio scoring, 
the team consists only of those who teach writing at the accountable grade level, and the 
composition of the team has changed minimally over the years (frequently only as teachers 
move from one grade level to another or retire). 

Example: One of the schools we visited has a scoring team consisting of all the fourth- 
grade teachers. Students at all grade levels (primary and upper elementary) are required 
to keep portfolios, and each grade-level team of teachers scores its own students’ 
portfolios. Teachers at the non-accountable grade levels are trained by the grade-level 
scoring team, after it has had updated training each spring. 

Low — Only accountable grade level teachers score, and they do so in isolation. 

F. Evidence of ongoing mentoring and informal professional development 

High — Teachers are members of formal teams (subject-matter teams, grade-level teams, 
primary “families,” special purpose teams) that are given adequate time for meeting and 
planning. In addition, networks with teachers outside the school are encouraged through 
sending teachers to appropriate professional development opportunities at regular intervals 
(e.g., professional association annual meetings, summer institutes, and the like), providing e- 
mail access for electronic networking, providing release time if needed to network with peers 
outside the school, etc. Informal networking and mentoring within the school is also 
encouraged and celebrated. 

Medium — Teams of teachers within the school are given common planning time, but the 
teachers report either that it is inadequate for substantive work or that they do not make 
consistent use of it. Networking with teachers outside the school is possible if a teacher 
persists in following up opportunities, but is not a school priority. Informal mentoring and 




113 



I 



7 



Srating.g 

networking within the school are tolerated but not encouraged. 

Low — There is minimal communication between teachers and administrators, mostly formal 
communications concerning the school. Teachers seldom talk among themselves about 
issues of teaching and learning, and any discussion of students’ writing is at a superficial 
level. 

G. Level of strategic collaboration in professional development — This indicator refers to 
degree to which the school faculty has participated together and support one another in 
professional development experiences linked with identified school needs. 

High — Teachers show awareness of the professional development plan and its connection 
with school weaknesses; most have had a part in developing the plan. Professional 
development is often school-wide. Teachers who attend conferences or workshops beyond 
school-based professional development attend as representatives of their school and are 
expected to bring learnings back to their colleagues. Teachers describe instances of giving 
and receiving help from one another. 

Medium — Teachers are aware of the professional development plan but most were not 
involved in developing the plan and they appear to know the specifics only so far as they are 
directly affected. Collegiality is encouraged, and there is informal collaboration (self- 
selected groups who share), but these groups appear to be mutually exclusive and not 
everyone is included in one. There is no formal mechanism— or encouragement— for teachers 
to support one another in professional development experiences. 

Low — Decisions about what professional development is needed are commonly made 
individually, and teachers may be unaware of the specifics of the professional development 
plan developed at the school. When teachers are trained together, they are in role-alike 
and/or grade-alike groups. Teachers are generally unaware of their colleagues’ level of 
training or expertise, and they are not encouraged or required to share what they learn in 
workshops with other teachers. They do not mention instances where teachers have helped 
one another and they do not appear to link identified school needs with professional 
development. 

3. Coordination across grade and subject areas 

Since writing is an area in which students should be making steady, ongoing improvement, 
coordination of the writing program across grade and subject areas is an asset to the program (but 
is not the norm). 

High — In departmentalized schools, teachers from different subject area departments share in 
the same professional development on writing and compare notes on a regular basis to ensure 
that the writing program is implemented in a planned, consistent manner to produce 
integrated, well designed writing instruction. In all schools, teachers from adjacent grade 
levels compare notes on a regular basis to ensure that writing instruction is consistent as 
students progress from grade to grade and that student progress in writing is well' documented 
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as they move from grade to grade (for instance, through the development of a cumulative 
portfolio). Feeder schools provide the schools that students move on to with adequate 
information about each student’s progress in writing and meet to ensure that the writing 
program at the higher level school builds on the program at the feeder school. 

Medium — In departmentalized schools, teachers from different subject area departments are 
encouraged but not required to participate in the same professional development on writing 
and to confer with one another periodically about the way writing is used in each subject area. 
Some teachers do this, but others do not. Similarly, teachers at adjacent grade levels may or 
may not coordinate their writing programs; usually some effort is made to ensure that student 
writing moves with the student from grade to grade, but the effort is not always successful. 
Teachers at the accountable grade levels are usually the most conscientious about 
coordination with other teachers. 

Low — ^No effort is made to coordinate writing instruction across subject areas or across grade 
levels. Individual teachers may express frustration at the lack of coordination. 

4. School climate/communication 

This indicator includes any evidence that relationships among administrators, teachers, students, 
and parents are friendly, collegial, and facilitate the development of the writing program. 

( 

High — ^Frequent, relaxed and cordial communication occurs among administrators, teachers, 
aides and support staff; with evidence of mutual respect and liking among faculty, staff and 
students, as well as any parents who may be present; with perhaps evidence from school 
displays and conversation that writing is valued as a worthwhile and enjoyable activity; teams 
of teachers who clearly work well together are another sign of a positive school climate. 

Example: An elementary school we visited has a large number of students whose parents 
transport them to school and some students were consistently tardy. Now the school starts 
every morning with an assembly to get the day off to a happy, enthusiastic start. Teachers 
are responsible for planning the assembly, which generally features either a speaker from 
the local community or students who have done something worth exhibiting. If students 
are late, it does not disrupt instruction, but students are now pressuring their parents to get 
them to school on time because they enjoy the assembly, and parents occasionally stay to 
attend the assembly with their children. The school has an extremely diverse student 
population, both ethnically and economically, but the administrators and staff reach out to 
all students and all parents in ways that appear to make all feel welcome and included. 

Medium — Cordial but businesslike relationships exist among administrators, teachers, staff, 
and support aides, with little emphasis on teamwork or meeting the needs of the “whole 
child”; there may be an emphasis on academic excellence but not on teamwork to produce 
academic excellence. 
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Low — Hallway, lunchroom and teachers’ lounge conversations are generally unrelated to 
instruction; minimal and formal communication between teachers and administrators, with 
support staff always clearly subordinate to teachers; an emphasis on order (and usually 
silence) in the hallways and in the classrooms, with exchanges between adults and students in 
non-classroom settings primarily directive rather than social. Hall and classroom displays 
appear to be mostly adult-created (or “cookie-cutter” student work), with little apparent value 
given to student writing. 

5. Evidence of communication with families and family support of the writing program. 

High — The school recruits and trains volunteers to help in the classroom (for example, to 
free teachers for conferencing, to read and give feedback on student writing, to share their 
own writing, to talk about the kind of writing necessary in their jobs.) In “family night” 
sessions, through newsletters, or through formal training, the school provides information to 
families about writing levels expected of students and ways parents can help at home. A 
conscious effort is made to help parents feel comfortable in the school. 

Example: An elementary school we visited provides classes for parents through the library 
program. The first class provided last fall was instruction in the writing process and 
appropriate ways for parents to work with children at home on writing. At the same 
school, one of the children we interviewed described how she herself had instructed her 
mother in appropriate conferencing techniques. 

Medium — The school provides written information to parents about the writing process and 
suggests ways they can help students at home with writing assignments, but does not actively 
recruit parents to help in the classroom. Individual teachers may encourage parents to 
volunteer in the classroom or give them pointers on how to work with children at home on 
writing. 

Low — Any communication to parents about the writing process stresses what they should not 
do (e.g., make suggestions that usurp the child’s ownership of the writing) rather than active 
ways they can help. Parents are not encouraged to volunteer in the classrooms, although a 
few active parent volunteers may be tolerated. No effort is made to explain to parents how a 
child's writing skills are progressing. 

6. Evidence that the school uses community resources in the writing program 

High — ^The writing leader actively seeks resources for such programs as having a writer visit 
the school for a period and work with students on writing as well as explaining how he/she 
makes a living as a writer. Individual teachers are encouraged to invite into the classroom 
people in the community who use writing in interesting ways in their adult careers. Such 
people may also be asked to judge school writing competitions or projects. Community firms 
or businesses may be asked to support the school's writing program in various ways (for 
instance, exhibiting student writing or providing prizes for a contest). 
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Example: Each year at a small town elementary school the fourth-grade students organize 
a blood drive for the local health department. They make all the arrangements for a blood 
drive at the school, and then they write persuasive pieces to convince members of the 
community to participate. Before the drive, they parade through their small town leaving 
these persuasive pieces at each store, to be distributed to the patrons. They also talk to 
their parents and neighbors about the blood drive. The blood drive requires expository 
writing, describing how to organize a blood drive, it leads to research on blood types, the 
need for blood, and how the blood will be used. Some students write personal narratives 
or fiction based on the research they have done. 

Medium — Individual teachers seek out community resources to enrich the writing program 
from time to time, but it is not a school priority. 

Example: The writing leader at a small, rural elementarj' school invited the school 
superintendent to visit the school and talk to the students about the many ways in which 
writing had played a part in his career. She wanted to find some way to support a writer in 
residence program but, to date, the visit by the superintendent was the only effort made at 
the school to use community resources to enhance the writing program. 

Low — ^There is no evidence that community resources are used in the school's writing 
program 

7. Focus and intensity of writing instruction programs 

This indicator requires a global assessment of the degree to which writing and/or writing 
portfolios are a clear school priority. 

High — There is abundant evidence that the school communicates to students the importance 
attached to writing. Student writing is prominently displayed in the hallways and in 
classrooms, and classroom walls display posters emphasizing various aspects of writing 
instruction/portfolio development. Interviews with administrators, teachers, and students are 
consistent in revealing the importance placed on writing at the school — and interviewees 
agree that writing well is an important life skill. If events outside the control of the school 
administrators interfere with instruction, an effort is made to ensure that it does not interfere 
with students’ opportunity to work on writing. 

Medium — Writing is a clear school priority, but there is a lack of focus in the writing 
program. Students do not articulate the importance of writing as clearly as teachers and 
administrators do, or they may reveal ignorance of strategies that teachers report using in 
writing instruction. There is a clearer focus on portfolio development than on writing as a 
valuable part of education, and writing may be a focus only part of the year. Teachers may 
draw a clear distinction between writing assignments meant for the portfolio and other 
writing assignments. Teachers are ambivalent about the value of writing, on the one hand 
considering it valuable but on the other hand resenting the time it takes away from other 
subject matter to do process writing (since writing is not well integrated into instruction). 

Low — Writing appears to be stressed only at the accountable grade levels and only for the 
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purposes of developing writing portfolios as part of the state assessment program. Teachers 
and students resent the demand to assemble writing portfolios and many dislike writing. 

8. Use of The Writing Portfolio Teacher's Handbook, 2nd Edition as a resource by all 
teachers teaching writing 

This indicator is strongly related to the overall priority placed on writing at the school and the 
leadership abilities of cluster leaders and other writing leaders in the school. If a school uses a 
district-produced Handbook rather than that produced by the state Writing Project, it should be 
examined or described well enough so that a judgment can be made whether it is or is not 
equivalent. 

High — Every teacher in the school received the Handbook and guidance in using it as part of 
professional development, and the writing leader(s) in the school continue to work with 
teachers to demonstrate productive uses for the Handbook. Teachers at all grade levels and in 
all subjects report using the Handbook consistently, though they may turn to the Handbook 
only during certain parts of the year (for instance, when introducing students to the. 
benchmarks for the various performance levels or when preparing to score portfolios). If they 
do not turn to the Handbook frequently, they report that they are familiar enough with the 
content so that they do not need it for general reference purposes any more. Students exhibit 
familiarity with the benchmarks and the criteria for the various performance levels. 

Medium — All portfolio-accountable teachers have copies of the Handbook and use it as 
described above, and a few other teachers who have a particular interest in the writing process 
may have been able to get a copy. Other teachers have guides to teaching writing that are not 
as extensive as the Handbook (often produced by a local writing leader) or have gained their 
knowledge of portfolio development entirely from professional development workshops. 

Some teachers are not aware that the full Handbook exists. 

Low — Only the teachers who are on the portfolio scoring team have copies of the Handbook, 
and they refer to it only when preparing to score portfolios. 

9. Evidence that teachers and students focus on writing 

A. Evidence that developing competent writers, not just portfolios, is the focus of the writing 
program. 

It is to be expected that any school in Kentucky will focus on completing writing portfolios. 
There should also be a focus on writing for its own sake or as an important part of instruction. 
A key criterion here is the degree to which portfolio writing is integrated into classroom 
instruction or is seen as separate from classroom instruction (inevitably interfering with other 
instruction). 

High — Writing is part of instruction at all grade levels and in all subject areas. Writing to 
learn and writing to exhibit learning are stressed as valuable, in addition to portfolio- 
appropriate writing. Writing assignments are not differentiated into portfolio assignments and 
other assignments, and potential portfolio pieces may come from a wide variety of sources, 
including' 
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writing the student has done for non-school related purposes. Students are exposed to a 
variety of ways in which writing is needed for adult life and in vocational areas. Teachers are 
conscious of developing their own writing skills and of the importance of students’ enjoying 
writing. Both teachers and students can describe their progress as writers. 

Medium — Writing is part of instruction at all grade levels but some subject matter teachers 
resent any demand that they incorporate writing into their instruction and refuse to do so. 
There is more emphasis on writing in the accountable grades and during the time that 
portfolios are being prepared than at other times, although students do some writing 
throughout the school year and in every grade. Although students speak of themselves as 
writers and often write when they are not required to for school assignments, teachers 
generally do not write except when they need to for clearly defined purposes. 

Low — Writing is neglected in the school curriculum except for the preparation of writing 
portfolios — and sometimes for practice in answering open response questions. Writing 
assignments are clearly identified as portfolio assignments and are not integrated into 
instruction. 



B. Evidence that teachers require more challenging work from students than they once did 
because students come better prepared 

High — When asked to account for the improvements in student writing, most teachers 
mention that writing has been well taught in previous grades so they are able to work with 
students at a much higher level than previously. Both teachers and students describe particular 
assignments that require more than students previously would have been asked to do at that 
grade level. 

Medium — Teachers report that student writing has improved over the years and that they now 
require more of students than they used to, but add that some students have always had the 
capacity to do the work now required and that some students come to them still lacking the 
skills to do the work they now require. 

Low — Teachers report that teachers at previous grade levels have not done a good job of 
teaching writing and that this hampers portfolio development because they must spend time 
working on basics that the students should have learned previously. 



C. Evidence of students’ confidence that they can eventually reach proficiency 

High — ^All students interviewed at the school report that the portfolio standards are not too 
high for all (or most) students to reach — though they require hard work from most 
students. Students may add that not all students are willing to put forth the effort required, 
but they consider themselves good writers and report that they are making the effort 
personally. Teachers and administrators agree that most students — over time — should be 
able to reach the standards and feel responsible for helping them do this. (If teachers see 
instruction as having impact on the quality of students’ writing to become proficient, this 
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probably indicates that the school is higher than “medium” on the criterion, though not at the 
top of the scale.) 

Medium — ^All or most of the students interviewed at the school report that the portfolio 
standards are not too high for all (or most) students to reach — though they require hard 
work from most students. Students may add that not all students are willing to put forth 
the effort required, but they consider themselves good writers and report that they are 
making the effort personally. Most teachers have reservations about the ability of some 
students to become proficient, either because they lack the ability (some have writing 
talent and some don't) or because they lack motivation or and adequate home environment. 
(Teachers may give other, less self serving reasons: e.g., the schools will never have 
adequate support or time to bring all students to the proficient level; or they may criticize 
the performance level itself, e.g., “proficient” is developmentally inappropriate at my 
grade level.) 

Low — Students express doubt that they will ever become proficient or say that a number 
of other students will not be able to do so. Teachers have major reservations of the sort 
described under “Medium.” 

10. Evidence that students routinely use and can describe the steps in the writing process 

A. . Students are familiar with and use the writing process 

High — All students interviewed name the essential steps of the writing process, of ten 
before they are asked to do so, and discuss cogently their importance in the students' 
development as writers. 

Medium — Some of the students interviewed are highly aware of the steps of the writing 
process and can discuss their importance, while others must be asked fairly specific 
questions before they discuss them. Most students, however, exhibit some awareness of the 
various components of the writing process and have gone through them with at least some of 
their teachers. 

Low — Students fail to discuss some steps of the writing process at all (for instance, pre- 
writing or publication) and generally seemi unaware that the various components of the 
process are important to help them develop as writers. 

B. Students are aware of criteria forjudging portfolios (and ways in which they learned these 
criteria) 

High — ^All the students interviewed report that each year one or more teachers reviews 
the criteria with them, going over and discussing benchmarks or particular pieces of 
writing that exemplify a particular performance level. Students are given written 
material reminding them of the criteria for each performance level, and the criteria are 
displayed or reviewed periodically as portfolios are being prepared. Students may report 
using the benchmarks on their own initiative to assess their own and others’ work. 
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Medium — All students report having at least one teacher during the accountability year 
go over criteria for the various performance levels, or the criteria may be displayed but 
not addressed consistently. Students feel free to ask their writing teachers if they need 
any additional information. 

Low — Students are not introduced to the criteria at all or are presented them in such 
simplified form that they are not really sure how their writing pieces are going to be 
judged. 

C. Use of computer to facilitate writing 

High — ^The school has sufficient computers so that students may have access to them at 
all stages of the writing process. Students have learned keyboarding skills at an early 
age, have access to the Internet at school and use computers to find information; they 
feel comfortable working on computers. Students have been encouraged to use the 
computer at all stages of the writing process and have learned from experience that this 
improves their writing, so most of the students interviewed choose to use the computer 
at all stages, even if they do not have computers at home. 

Medium — The school has sufficient computers so that students may have access to them 
at all stages of the writing process, but this is a recent development and not all teachers 
or students have gotten used to it. Students who have computers at home still are more 
comfortable using the computer than those who do not and are more apt to use the 
computer at all stages of the writing process, but this is begirming to change, and the 
teachers are aware that students would benefit from greater use of the computer in 
writing. 

Low — The school either does not have enough computers for all students to use them at 
all stages of the writing process or they are not yet sufficiently accessible to students. 
Teachers are generally unaware of the advantages of using the computer at all stages of 
writing so do not encourage students to do so. Students who have computers at home 
are at an advantage in using computers for writing than students who do not. Computers 
at school are used mostly to produce final copy, and some students may not have access 
to them even for that purpose. 

D. Use of feedback to improve writing (conferencing with teachers, family, or other 
students 

High — ^Teachers and students both report that students conference with peers as well as 
with teachers as they revise their writing pieces and that the feedback they receive from 
both sources is helpful. Students show no reluctance to share their writing with peers, 
even students they may not know well; they say that students at the school all know what 
sort of questions are helpful. Students share their writing with family members and 
friends outside of school and ask for their feedback (teaching them how to ask helpful 
questions, if necessary). Neither teachers nor students report problems that prevent 
students from receiving as much feedback as they need during the writing process. 
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Students clearly feel ownership of their writing. 

Example; Teachers at one elementary school we visited taught primary 
students appropriate methods of conferencing and then made it clear that, 
since teachers and aides were not always available to conference with 
students, students who needed feedback were to find other students at about 
the same stage of composition and request a conference. Before they entered 
fourth grade, students had developed the habit of independently seeking 
feedback from other students as they worked on writing pieces. It was 
understood that they were supposed to do this, and they had the freedom to 
seek quiet places for the conferencing, including working quietly in the 
hallways. 

Medium — Teachers and students report that students are expected to seek feedback 
from other students as well as teachers, but most students exhibit some reluctance to 
share their writing during intermediate stages. They may report avoiding conferencing 
with other students or seeking feedback only from close personal friends or students they 
trust to give good feedback. Teachers as well as students report that students seem to get 
more out of conferencing with teachers than out of conferencing with peers. They may 
have developed systems to increase the amount of time available for students to 
conference with adults (for instance, increase the amount of time instructional aides can 
spend working one-on-one with students) but still feel there is a problem finding enough 
time for conferencing with students. 

Low — Even if teachers report that students do peer conferencing, the students report that 
they are seldom required to do this and that they can successfully avoid it most of the 
time. Teachers have little time to conference with students when they need help at 
various stages of the writing process, and they have not developed systems for increasing 
the amount of time instructors or aides are available to provide feedback to students. 
When they — or parents — do provide feedback, it tends to be primarily directive, so that 
students sometimes feel a piece of writing is no longer their work. 



11. Teacher writing 

Teachers who are not confident of their ovvm ability as writers will presumably have difficulty 
helping students become proficient. Teachers who write with students appear to have a powerful 
effect on students’ interest in writing and confidence that they can become proficient. 

A. Evidence that teachers write and share their writing with students 

High — All teachers interviewed at the school can describe instances when they temporarily 
abandon the role of “teacher” to write with students. Often this will be journal writing, but 
to rate a “high” rating on this attribute, teachers should be able to describe other occasions 
on which they write at the same time as the students — or when they complete a writing 
assignment they have given students and share their writing at the same time that students 
are sharing. 

Medium — Some of the teachers interviewed can describe instances when they write with 
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students — but this is usually journal writing only. Some teachers report that they occasionally share with 
students a piece of writing that is appropriate for a writing assignment the students have been given. 
Students describe these instances also — and may actually remember instances of teachers writing with 
them that the teachers themselves do not describe to the interviewer, but it is clear that such instances are 
occasional. 

Example: The power of teachers sharing their own writing with students was illustrated vividly 
during interviews with three ninth grade students in a rural school district in central Kentucky. 
When asked if they remembered any of their eighth grade teachers sharing writing with them in 
class, all three ninth graders (two girls and a boy) recalled that their eighth grade language arts 
teacher read a personal narrative that she wrote about her granddaughter. The students 
remembered the teacher’s narrative as being interesting. That particular teacher retired at the 
conclusion of the school year and much happened to the trio of students in their ninth grade (the 
interviews were conducted in May), still they quickly recalled the time she shared a personal 
piece of writing with them in the previous school year. 

Low — ^Neither teachers nor students describe any instances of teachers writing with students, other 
than writing on the chalkboard and requiring students to copy what they have written. 

B. Evidence that teachers introduce students to the many functions of writing in adult life 

High — Teachers report that they frequently talk with students about the ways they use writing in 
their everyday life and point out adult accomplishments that require writing. From time to time, 
they invite adults in a variety of occupations to the classroom to discuss the role of writing in 
their work. Students are aware of a large number of ways in which writing is needed in adult 
life and can describe them to the interviewer. 

Medium — Some of the teachers interviewed discuss specific instances in which they brought to 
students’ attention the need for writing in one adult role or another. On rare occasions, an adult 
may have been invited to the school to enhance writing instruction by talking about the role of 
writing in their life and work. Students are aware that they will need to write as adults and can 
describe to the interviewer at least a few ways they anticipate writing as adults. 

Low — ^There is no evidence from either teachers or students that there is any discussion in the 
school of the role of writing in adult life and work. 

12. Evidence of student choice of topics and formats for particular writing assignments 

High — ^All teachers interviewed at the school report that most assignments provide students with 
choice (appropriate to the subject area). Some assignments provide students with a 
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format and a range of subjects within a particular prompt, but others allow students to choose the 
format to address a particular topic or allow for choice of both format and topic. Students are also 
encouraged to add to their writing folders pieces they have written not for school assignments but 
for real world purposes outside school. Students agree with the teachers that they have many 
opportunities for choice of both format and topic in their writing and that most teachers give them 
opportunities for choice. 

Medium — Most teachers interviewed report giving assignments that specify format and the general 
topic area but allow students to make specific choices within the general area. Teachers may 
provide students with a list of possible topics or may brainstorm possible topics with the class and 
allow students to select from the list or choose a similar topic themselves. A few teachers report 
giving students a greater range of choices than this. Students agree with the teachers’ reports. 

Low — All teachers interviewed report giving students a very narrow range of choices (for instance, 
specifying format and giving student three possible topics to choose among). Many writing 
assignments allow for no choice at all. Students agree that they exercise very little choice in their 
writing. 

13. Evidence of real-w'orld writing 

A. Evidence of students having the opportunity to use writing for real-world purposes 

High — Teachers interviewed report that students have frequent opportunities to write to real 
audiences and that these communications are intended to have actual results (for instance, letters to 
the school board asking for specific improvements in the building or the lunch program, or 
pamphlets — later delivered — explaining the school's rules to an entering class). Students tell 
detailed stories about this type of writing, including the purpose for writing, what they wrote and 
why, and what the response was. 

Example: In one elementary school we visited, students were assigned to write to their parents 
prior to Christmas to ask for a particular present the child wanted very much. The letters were 
mailed to the parents, and the children were able to tell the interviewer how his/her parents had 
responded to it. 

Medium — Teachers report a variety of assignments to write to real audiences, but the letters are 
generally informational and not intended to produce particular results (for instance, letters to pen 
pals in other states or other countries). Teachers may also report a variety of assignments to wTite 
for imaginary audiences and may not differentiate sharply between the imaginary and the real 
audiences, understanding both as “real world writing.” Students’ accounts agree with those of their 
teachers, except that students are normally quite aware which audiences are imaginary and which 
are real. 
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Low — ^Neither teachers nor students mention writing for real world purposes to actual audiences at 
all, and they do not describe such writing even when specifically asked about it. Teachers either do 
not understand the term “real world writing” or describe writing for imaginary audiences when 
asked about it. 

B. Evidence that students are aware of the audiences for which they are writing and that the audiences 
are important to them 

High — Students discuss audience when asked to describe the writing process, and they differentiate 
clearly between imaginary or abstract audiences and actual audiences. They describe clearly what 
real audiences they have addressed in their writing, how they shape their writing to the audience, 
and why they think it’s important to communicate with those audiences. They report that teachers 
expect them to incorporate pieces they have written to real audiences for real world purposes in 
their writing portfolios. 

Medium — Most students talk about audience only when prompted by the interviewer, but it is clear 
that they do write at least occasionally to audiences who read what they write. Students may or may 
not be able to describe the audiences’ reaction to what they wrote and may or may not think it is 
important to communicate with those audiences. 

Low — ^Neither teachers nor students report writing to real audiences as part of the writing program. 
Teachers may describe imaginary audiences in response to questions about real audiences. 

14. Evidence of integration of writing instruction into general instruction 

A. Teachers using reading as a source of ideas and models for writing for a variety of purposes 

High — ^Teachers and students who are interviewed consistently talk about reading (in language arts 
or any of the other subject areas) as models for writing in that area and as a source of ideas for 
formats and topics. 

Medium— Some teachers and students who are interviewed can talk about particular ways in which 
classroom reading has served as a source of ideas for formats or topics or as a model of a type of 
writing. 

Low — ^Teachers and students discuss writing as totally separate from reading and do not draw the 
connection even when asked about reading as a source of ideas for writing. 
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B. Language arts teachers teach mechanics in context of writing 

High — Teachers and students who are interviewed report that mechanics are taught in the context 
of writing, primarily in mini-lessons designed to fit writing needs as they are identified. Students 
speak of improved spelling and punctuation as part of the overall writing process. 

Medium — Teachers report that they are trying to fit mechanics in the content of writing, but what is 
taught usually is determined by the textbook rather than by student needs. They report reinforcing 
these lessons in the context of particular writing assignments or stressing in the lessons mechanics 
they know will be important to upcoming writing assignments. 

Low — Teachers report teaching mechanics (spelling and grammar) as separate subjects. In teaching 
mechanics as drill, they follow the order of the textbook and do not make a special effort to teach 
mechanics with which students are having difficulty in their writing assignments. 

15. Evidence that students receive consistent messages, consistent with Kentucky Writing 
Program principles, from different teachers 

High — Teachers interviewed at the school talk about the writing program as an integrated whole, 
even if teachers in different subject areas have different contributions to make to it. Their 
description of the objectives and progress of the writing program at the school is essentially 
consistent from teacher to teacher and consistent with the way students describe the writing 
program. 

Medium — A few teachers (usually language arts teachers) take the lead in implementing the wTiting 
program and have a clear vision among themselves but the vision is not shared with other teachers; 
teachers in different subject areas or at different grade levels may have very different conceptions of 
the school’s writing program. Students report different understandings of the school’s writing 
program, depending on the teachers they have, or their account of writing instruction differs from 
their teachers’ accounts. 

Example: In a high school we visited, two language arts teachers appeared to be largely 
responsible for the school’s success in improving portfolio scores. Their students, when 
interviewed, were articulate about the various steps in the writing process and described 
challenging, interesting writing assignments, while students of other language arts teachers 
described only “cookie-cutter” assignments and appeared unaware of the writing process as a 
whole. 

Low — Students may receive consistent messages about writing, but the messages are that writing is 
not a school priority, and few or none of the faculty teach the writing process in the ways 
recommended by the Kentucky Writing Program. 
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16. Consistency among student/teacher/administrator perspectives 

This indicator requires a global judgment about the degree to which students, teachers, and 
administrators told the same story in response to the same or similar questions. 

High — There were virtually no inconsistencies in what students, teachers, and administrators had to 
say; all described the writing program in the same way, discussing a variety of writing assignments 
and telling revealing anecdotes about writing instruction. 

Medium — Students, teachers, and administrators described the writing program in generally similar 
terms. There were inconsistencies, but for the most part they did not involve key indicators or add 
up to a grossly inconsistent story. If administrators have a different perception of the writing 
program, but teachers and students have consistent perceptions, the school is probably in the 
medium range on this indicator. 

Low — Descriptions of the writing program by students and teachers are so different they might 
have been talking about different schools. Administrators may tell the same basic story as the 
teachers, or they may have a different perception of what is going on than the teachers. 




127 



APPENDIX G: 



Rating Form 




128 



3rating. f 



1 



SCHOOL STUDY OF WRITING INSTRUCTION 

Directions for completing rating form 



Before filling out the form, review the school report. Your judgments about some, if not 
most, of the indicators will come from your assessment of the cumulative information as 
well as from information about a specific indicator. To make judgments, you need to look 
at the range expressed, particularly at the degree to which students and teachers agreed in 
their understanding of the writing program. Widely disparate responses between role 
groups (administrators and teachers, teachers and students) alert you to areas that need 
closer scrutiny. 

The response requires marking a point on a sliding scale with an “x” to indicate the degree 
to which your school typifies an indicator. Mark any point on the line except the vertical 
(dividing) lines. It is tempting to mark all categories “average,” so avoid the exact center of 
the line unless you are convinced the school is absolutely average for that indicator. Also, it 
is very unlikely that the school would warrant the same “mark” for every indicator, so try 
to make independent ratings for each indicator. Where you have insufficient information 
to make an assessment, check the line so indicating. 

The indicators include descriptions of high, medium, and low performance. These 
descriptions should serve as guides, not as criteria that must be present to justify a 
particular rating. To illustrate this, we have added some slightly offbeat descriptions of 
schools visited as this study process was developed. 



REMEMBER: This analysis is intended only to help you strengthen your school’s writing 
instruction. No one outside the school sees it. Be honest in your assessment. 
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SCHOOL STUDY OF WRITING INSTRUCTION 
RATING FORM 

Note: Use rating guide for explanation and examples of the indicators. 

School Date: 

1. Administrative support 

A. District level 

Insufficient Information Lo w ////// High 

B. School level 

Insufficient Information Low ////// High 

2. Availability and use of quality professional development/technical assistance 
A. For writing leaders 

Insufficient Information Low ////// High 



B. For portfolio accountable teachers 
















Insufficient Information 


Low 


/ 


/ 


/ 


/ 


/ 


/ 


High 


C. For other teachers 


















Insufficient Information 


Low 


/ 


/ 


/ 


/ 


/ 


/ 


High 


D. Attention to writing that is appropriate for content areas other than language arts 


Insufficient Information 


Low 


/ 


/ 


/ 


/ 


/ 


/ 


High 


E. Teacher training and participation in portfolio scoring 










Insufficient Information 


Low 


/ 


/ 


/ 


/ 


/ 


/ 


High 
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F. Evidence of ongoing mentoring and informal professional development 

Insufflcient Information Lo w ! II II ! H igh 

G. Level of strategic collaboration in professional development 

Insufflcient Information Low ! II II ! High 

3. Coordination across grade and subject areas 

Insufflcient Information Low ! II II ! H igh 

4. School climate/communication 

Insufflcient Information Low ! II II ! H igh 

5. Evidence of communication with families and family support of the writing program. 

Insufflcient Information Lo w ////// H igh 

6. Evidence that the school uses community resources in the writing program 

Insufflcient Information Low ////// H igh 

7. Focus and intensity of writing instruction programs 

Insufflcient Information Low ////// High 

8. Use of The Writing Portfolio Teacher's Handbook, 2nd Edition as a resource by all 
teachers teaching writing 

Insufflcient Information Low ////// High 

9. Evidence that teachers and students focus on writing 

A. Evidence that writing is the focus of the writing program, not just portfolios 

Insufflcient Information Low ////// H igh 

B. Evidence that teachers require more challenging work from students than they once 
did because students come better prepared 

Insufflcient Information Lo w ////// H igh 
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C. Evidence of students’ confidence that they can eventually reach proficiency 

Insufficient Information Lo w ////// High 

10. Evidence that students routinely use and can describe the steps in the writing process 

A. Students are familiar with and use the writing process 

Insufficient Information Low / / I I / / High 

B. Students are aware of criteria for judging portfolios (and ways in which they learned 
these criteria) 

Insufficient Information Low ////// High 

C. Use of computer to facilitate writing 

Insufficient Information Low ////// High 

D. Use of feedback to improve writing (conferencing with teachers, family, or other 

students 

Insufficient Information Low ////// High 

11. Teacher writing 

A. Evidence that teachers write and share their w riting with students 

Insufficient Information Low / I I / / / High 

B. Evidence that teachers introduce students to the many functions of writing in adult life 

Insufficient Information Low ////// High 

12. Evidence of student choice of topics and formats for particular writing assignments 

Insufficient Information Low ////// H igh 

13. Evidence of real-world writing 

A. Evidence of students having the opportunity to use writing for real-world purposes 
Insufficient Information Low ////// High 
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B. Evidence that students are aware of the audiences for which they are writing and that 
the audiences are important to them 

Insufficient Information Low ////// High 

14. Evidence of integration of writing instruction into general instruction 

A. Teachers using reading as a source of ideas and models for writing for a variety of 
purposes 

Insufficient Information Low ////// High 

B. Language arts teachers teach mechanics in context of writing rather than separate 
from writing 

Insufficient Information Low ////// High 

15. Evidence of consistency of messages students receive about writing from different 
teachers 

Insufficient Information Low ////// High 

16. Consistency among student/teacher/administrator perspectives 

Insufficient Information Low ////// High 
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APPENDIX I: 



Findings by Field-Test School 



FINDINGS BY FIELD-TEST SCHOOL 



The following discussion is an expanded account of each field-test school’s experience in 
conducting the School Study of Writing Instruction. The 5 facilitated schools were each assigned a 
facilitator to guide them through the process. The 6 independent schools were each assigned a contact 
person to help them via telephone. 

Condition 1: Facilitated Schools 



Facilitated School #1 

Description of FSl's study process. FSl completed its school study in 9 weeks during an 1 1- 
week period between February 24 and May 13,1 999 (the study was suspended for a week each of spring 
break and state testing). The assistant principal of FSl began calling the facilitator in December, asking 
when they could get started. Approximately 8 weeks elapsed from the time of this first contact until 
they started on February 24; FSl would have begun on February 5 but agreed to postpone to 
accommodate the schedule of another field-test school who needed the facilitator. The facilitator 
reported that the assistant principal and Steering Committee of FSl possessed a wonderful attitude 
about conducting a self-study of their writing program, willing to do anything to improve it. She said 
that they probed deeply for information because they wanted an accurate portrayal of their writing 
program. Also, they took the initiative of including input from their feeder school in their self-study. 
The Committee contacted the facilitator few times, more often to verify scheduling of steps in the 
process and to communicate the school’s progress than to request assistance. Most contacts between 
them were via e-mail. The Committee did call once about a problem receiving the stipend. The 
facilitator stated that the Steering Committee could have done the study without her assistance because 
of the interest level of the assistant principal and the other Steering Committee members and, 
particularly, the assistant principal’s interest in progress for all grades and content areas. She stated, 
“They really didn’t need me.” 



The Steering Committee consisted of 5 members: the assistant principal, 3 faculty, and 1 parent. 
The facilitator’s log described the Steering Committee’s preparedness at the various steps in the process. 
At the first meeting with the facilitator, the Committee reviewed the School Study handbook. 
According to the facilitator, the Committee had begun before she arrived, during which time the 
assistant principal had covered the first section of the handbook with the rest of the Committee. She 
also observed that the attitude of the Steering Committee was positive. In a phone interview, the 
assistant principal indicated that the facilitator had presented the handbook well. Also, he stated that 
the handbook was systematic, easy to follow, and needed — they used everything in it. 



Eight teachers and 1 2 students were interviewed in 2 consecutive days, teachers one day and 
students the next. Included in the teacher sample were two teachers from FS 1 's feeder school, who were 
interviewed via written survey instead of verbally. The assistant principal interviewed teachers because 
it was convenient in terms of scheduling. Two parents, one of them on the Steering Committee, 
interviewed students in groups of 2-3, except for one individual interview. In a phone interview, the 
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assistant principal said that interviewing students took more time because it was harder to extract 
information from them, but it was doable within a 51 -minute class period. The assistant principal, 
rather than the principal, completed the principal inter/iew (in written survey form) since he was more 
familiar with the writing program. It is unclear whether a district administrator was interviewed; it 
appears as if the assistant principal also completed this interview (also in written survey form). The 
Steering Committee chair aggregated all teacher responses onto one form. Student responses were not 
aggregated; however, as most were interviewed in groups, responses of interviewees in the same group 
were recorded on one form. None of the interviews were typed. 

The Steering Committee members chose to write the report in a meeting, rather than 
individually. At the request of the facilitator, they held the meeting at another school’s computer lab. 
FSl participants were prepared to write when they arrived on March 15, having read the completed 
interviews beforehand. According to the assistant principal’s feedback, this meeting lasted 3 hours and 
was the longest of all their meetings. The report was typed during this meeting as it was being 
composed. 

The assistant principal intended that all faculty should be at the rating meeting, which meant that 
it had to be held during school hours and arrangements made for student supervision. The meeting was 
originally scheduled when students would be at an outdoor event; however, it was delayed for 10 days 
due to a district mandate that students not be outside in the wake of the Columbine High School 
shooting in Littleton, Colorado. Although the facilitator had intended to be present, she was not 
apprised of the date of the rescheduled meeting until after it had taken place. 

The rating meeting lasted 2'/2 hours. The assistant principal reported that they had encountered 
difficulty reaching consensus once the small groups had reassembled into a large group. 

The Steering Committee of FSl had an original idea for the conclusion of the rating meeting. 
The Committee created a brief reflection form that it asked all participating faculty to complete 
individually. The form asked faculty members (1) what they had learned from the report and (2) what 
they had learned from the rating process. Eighteen faculty responded. 

In answer to question one, what they learned from the report, 1 8 responded, some with multiple 
responses. The most frequent response to question 1 Was the realization that teachers’ and students’ 
perspectives vary about what students are learning (7 responses). (They did not specify in what ways 
the perspectives varied.) The second most frequent response was the statement that both administrators 
and teachers are concerned about improving the writing program or student writing (4 responses). 
Several other responses concerned writing program needs that had surfaced: help for content area 
teachers outside language arts to participate in the writing program (3 responses), computers (3), better 
student understanding of the writing process (2), better student understanding of benchmarks (2), more 
time and teacher training for writing program improvement (1), and parent and community involvement 
( 1 ). 

In answer to question two, what they learned from the rating process, 1 8 responded, some with 
multiple responses. Four respondents said that they learned that the school was doing better in its 
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writing program than expected (i.e., “good,” “average,” or “doing better than anticipated”). Five said 
the same, but added that the writing program still needed improvement. Seven more respondents simply 
stated that the school’s writing program needed improvement. Of the total 12 respondents who 
indicated that the writing program needed improvement, 8 gave examples, the most frequently 
mentioned element for improvement being training and technical support for collaboration with content 
area teachers outside language arts (4 responses). 

Following the faculty rating meeting, both parts of step 4 of the study, setting priorities and 
planning actions, were assigned to the school’s Consolidated Planning Committee. Regarding 
completion of the school study, the facilitator expressed faith in FSl's ability to do so because of the 
leadership she observed in them. The assistant principal’s closing thoughts on the study process were 
that it was useful for discovering aspects of the writing program and it gave the faculty data which they 
would use, saying that the most beneficial outcome was the illumination of the program’s status at 
different grade levels. 

Principal and Steering Committee feedback forms from FSl. Following the completion of 
their school study, each of the 5 Steering Committee members returned a feedback form. In lieu of the 
principal feedback form, the assistant principal filled out the same Steering Committee feedback form 
as the others. The principal did not complete the principal feedback form because the assistant principal 
had been the administrator most closely associated with the School Study process. Of the 5 feedback 
forms received, 3 were missing the second page containing questions 5-9, because it had apparently 
been inadvertently overlooked in the copying of the form. 

In response to question 1 , what was the respondent ‘s personal time spent to conduct the study, 

1 respondent said 10 hours, 3 respondents said 8-10 hours, and 1 said 12 hours. In response to question 
2, which tasks in the School Study required the most time, 1 said participating in Steering Committee 
work and interviewing, 3 said the report, and 1 said both the report and interviewing. So the report was 
mentioned four times and interviewing twice. In the phone interview, the assistant principal reported 
that compiling the report was the most cumbersome task of the study, partly because the computer 
program that they had used was not compatible with the forms provided on disk. He felt that 3 hours 
for the report-writing meeting was too long. 

In response to question 3, how many Steering Committee meetings were held, the Committee 
held 4 meetings prior to the faculty rating session. Since one of them was the report- writing meeting, 
only 3 of these meetings would be considered specifically planning meetings. 

In response to question 4, whether they could have carried out the study without outside help, 
3 Steering Committee members said yes and 2 said no. To the second part of question 4, what tasks 
would have been difficult without outside help, 2 named the initial overview and setup of the study 
process, 2 named compiling the report, and 1 named both of these aspects of the study. 

Due to the copying error mentioned above, responses to questions 5-9 were given by only 2 of 
the 5 respondents. In answer to question 5, whether the school profile was an accurate picture of their 
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writing program ’s strengths and weaknesses, 1 judged it to be semi-accurate, but was not as sure being 
a parent, and 1 said that it was, because it illustrated problems that they had already suspected. In 
answer to question 6, if the outcome of the study was worth the effort, both said yes, 1 adding that they 
now had data to support future plemning. In answer to question 7, how the study had affected or would 
affect the school, I said through planning and changes to the writing program; and I said it had affected 
the school already by having identified problem areas at the feeder school, and, in the future, it would 
affect the school through professional development and moving writing responsibilities to all grade 
levels. In answer to question 8, which parts of the study process were most helpful, I said the student 
perspectives as discovered through interviews, and I said both the interviews and report, because of the 
comparison between the various role groups’ perspectives produced by these documents. In answer to 
question 9, which parts were least helpful, both respondents said none. 

Question 10 solicited suggestions for improvement to the study process. Two said none were 
needed and I respondent left it blank. One suggested simplifying some of the written procedural 
language and 1 suggested clarifying interview questions, as some of the interviewees’ answers were 
vague. 



Question 1 1 asked about the utility of the sample report as an aid to report writing. Four 
respondents said it was helpful and recommended continuing to include it in the School Study 
handbook. One said it was somewhat helpful. 

Reliability of FSl's report. FS 1 submitted interviews from all role groups, including 8 teacher 
interviews and 10 student interviews, plus a teacher interview compilation, and the report. Although 
student interviews were not compiled onto one form, responses of interviewees in the same group were 
recorded on the same form, each student’s responses designated by a number; therefore, four forms 
containing the responses from a total of 1 0 students were submitted. Although two separate forms were 
submitted for administrator interviews, it is possible that the assistant principal served as both the 
principal and district administrator interviewee. Finally, it should be noted that FSl omitted 2 topics 
from its report (topics 9D and 9E), so only 3 of the chosen 5 could be considered. 

The reliability examination revealed that FSl's 3 report responses were partially equivalent to 
the 3 corresponding items in the reviewer’s sample in terms of accuracy and breadth of the number of 
data points considered, but with less detail. Two contained descriptive detail, yet 1 did not have enough 
for adequately summarizing and characterizing aspects of the writing program. In 2 of the 3 topics, one 
possible major data point was omitted. The responses contained no misrepresentations of data. In sum, 
although the 3 report responses under scrutiny did contain a few helpful points of information, the 
responses were too brief, 2 containing only a one-sentence statement for one of the role groups. Finally, 
the “Other” space was not utilized in the 3 responses, but it was used in a few instances in the remainder 
of the report for recording additional information. 
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Facilitated School #2 



Description of FS2's study process. F S2 conducted its self-study in 5 weeks between February 
24 and March 31, 1999. Six and weeks elapsed from the time of the facilitator’s first call to the 
school in mid-January and the start of the study. The first meeting between the Steering Committee and 
the facilitator, originally scheduled for February 5, was postponed by FS2 on late notice until February 
24. According to the facilitator’s report at the May progress meeting and her log, FS2's school study 
began unfavorably for a number of reasons. First, after the principal greeted her, he did not participate 
in the study process. Second, the Steering Committee was composed of 8 people, which the facilitator 
thought was too many. Third, when she arrived at their first meeting, several Committee members did 
not know why they were there. Fourth, no one wanted to be the Steering Committee chairperson. The 
obvious choice would have been the writing leader; however, she had recently left the school. The 
position was forced on the Title I teacher due to her flexibility to be out of the classroom and the fact 
that she was now the writing leader. She reluctantly accepted the chairperson responsibility, according 
to the facilitator. 

The 8 -member Steering Committee was composed of the Title I teacher, language arts and other 
content area faculty, and 1 parent. The School Study progressed in a disorganized way, according to 
the facilitator. The facilitator initiated all calls and e-mail contacts. She reported that the Steering 
Committee chair never contacted her with questions or to request additional assistance. She did, 
however, receive a call from the district office, which was receiving bills from the school for the 
stipend; apparently, the school experienced a delay obtaining it. When the facilitator arrived for 
meetings, the Committee was unprepared. They would spend the. beginning of meetings copying 
sections of the handbook that should already have been copied in total, distributed to, and read by the 
Steering Committee members. Thus meetings typically began 30 minutes late. The HSE participated 
in the study process, which provided the Committee with a measure of support, according to the 
facilitator. 

The chairperson reported that the Steering Committee had originally scheduled a teacher to be 
the interviewer, but the substitute became ill and canceled. Therefore, the Steering Committee chair 
performed the teacher interviews (again, because she had no regular classroom). Eight teachers were 
interviewed. In preparation, the Committee had retained substitutes for the interviewees so that teachers 
would not be forced to use their plarming periods. An unusual technique she employed was to interview 
teachers in pairs, which was not intended or suggested in the handbook (the collaborative research team 
believed that individual teacher interviews were more conducive to teachers speaking freely). She 
accomplished the teacher interviews in 1 day, adding that it was especially difficult without a scribe. 

Two parents — one of them a Steering Committee member — conducted student interviews.'* 
The Steering Committee chair said that they chose parent(s) rather than school staff to be interviewers. 



'*The facilitator believed that one parent, with the HSE serving as scribe, performed the interviews since this was the 
arrangement decided upon in a meeting; however, the Steering Committee chair’s account is accepted for this discussion since 
she was the person most involved with the study throughout the process. 
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based on their assumption that parents would be more likely to elicit honest responses from students. 
Twelve students were interviewed in groups of 3 over the course of 1 day, the day before the teacher 
interviews. Interviewers were able to interview each student group comfortably within 1 class period. 

The facilitator expressed dismay about the interviews, feeling that interviewers did not probe 
beyond asking the questions once, despite the facilitator having practiced interviewing with Steering 
Committee members at their first meeting. The Steering Committee chair was displeased with the 
teacher interviews for a different reason: she felt that some teachers gave incorrect responses because 
they do not understand what writing is, i.e., “worksheets are not writing.” Thus the Steering Committee 
chair felt that to be a true study, all teachers, including those more knowledgeable of the writing 
program, should have been interviewed. On the other hand, she was pleased with student interviews, 
assessing their responses to be honest overall. However, she felt that some students did not understand 
the phrasing of the questions and that rephrasing questions so students would understand was difficult 
for interviewers to do. 

After interviewing teachers, the Steering Committee chair also accepted the task of aggregating 
the teacher interviews and student interviews onto one form each, reporting that “it took hours to 
collapse the interviews.” As writing portfolios were being scored during that time, she had to stay after 
school until 7:00 or 8:00 p.m. every day for a week to compile and type the two sets of interviews. 

FS2 wrote its report in a meeting of the Steering Committee in its computer lab. However, 
according to the facilitator, the computer lab was not adequately set up beforehand. Also, because the 
software was not compatible with that on the disk, report writers had to retype the headings on the 
report form as they worked. The report-writing meeting lasted 3 hours. According to the Steering 
Committee chair, because the report sections had been completed by various writers in three to four 
different software programs, she had to retype all sections of the report later to create the entire 
document. 

A Steering Committee meeting for the purpose of planning the faculty rating meeting, held 
shortly after the report-writing meeting, was disorganized like previous meetings in the facilitator’s 
assessment. The facilitator reported that several Steering Committee members still did not know their 
purpose for being there. Steering Committee members still had not been provided with a complete 
handbook; the section that the Committee should have read prior to this planning meeting was provided 
to members only at the start of the meeting. The meeting began 1 5 minutes late because some Steering 
Committee members arrived late and some had to leave the meeting to retrieve their copy of the report. 
Several had not read the report. The facilitator reported that, due to these circumstances, this meeting 
took much longer than necessary (slightly over 3 hours). 

The faculty-wide rating meeting was also poorly planned and progressed in a disjointed fashion. 
It lasted approximately 2 1/4 hours. The rating session began 30-45 minutes late because some faculty 
left to buy refreshments, others arrived late, and many came without their report copy and had to leave 
to find it. As had been the case for Steering Committee members during the study process, the 
facilitator observed that the faculty in general did not know the purpose of the rating meeting and had 
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not read the report beforehand. The principal attempted to explain the reason for the session; the 
facilitator then clarified the purpose more fully. The HSE attended the meeting and also spoke to the 
faculty, telling them that the School Study would be valuable for their future planning. The facilitator 
believed that she sensed negativity among the faculty. 

At the meeting, the rating results were displayed on enlarged version of the school profile, a 
method devised by the facilitator. The facilitator recorded several faculty impressions and comments 
generated during discussion and written reflections. In general, she said that the teachers were not 
surprised that administrators did not know what was happening relative to the writing program, as 
shown on the school profile. Also, teachers were surprised to discover the existence of a longstanding 
district policy requiring the development of writing portfolios in nonaccountable grades. 

Fifteen specific comments from faculty discussion were recorded by the facilitator in her log, 
revolving around a few themes. Five comments were reactions to the rating results in general or to a 
specific aspect of them, 2 of which questioned the validity of report and rating results as an accurate 
depiction of their school’s writing program, e.g., ‘“The way I see things at this school are very different 
at times than what the survey shows.’” Three comments reflected faculty’s perceived need for 
professional development or doubt about personal ability to effectively teach writing, e.g., ‘“We are a 
novice overall so how can we develop distinguished writers? I’m novice so that’s why I don’t teach 
writing as such [at the distinguished level].”’ Three reflected a desire to help students, e.g., “‘We need 
a more positive attitude to benefit our students. I hope the results will really help them.’” Three 
reflected concern that the timing of the study was poor for having a meaningful influence on the writing 
program, saying it should have been conducted earlier in the school year. Finally, one comment was 
idiosyncratic. 

Following the rating meeting, the facilitator participated in a debriefing with the Steering 
Committee. Several members expressed a wish for more time to have completed the study. They felt 
rushed due to other school demands. One individual reiterated the belief expressed in the faculty 
meeting that interview responses did not faithfully represent the school’s actual practices in carrying 
out the writing program, but she/he did not specify in what ways the responses were unrepresentative. 

The Steering Committee told the facilitator that they planned to assign step 4, setting priorities 
and actions, to another committee. Sometime after the rating meeting, the HSE left his post at the 
school. The facilitator stated that minus HSE support, she is dubious about FS2's willingness to 
progress to step 4, despite the intention to designate a committee to set priorities and actions. She was 
not abreast of FS2's progress after the rating meeting. In a phone interv’iew with another member of the 
research team, the Steering Committee chair reported that FS2 would be waiting until the 1999-2000 
school year to do step 4 and did not mention having designated a committee. 

Principal and Steering Committee feedback forms from FS2. The principal completed a 
feedback form approximately 3 months after the completion of the study. In response to question 1 , 
whether the writing program profile was an accurate picture of FS2's program’s strengths and 
weaknesses, he said yes, because it was written by a group of teachers rather than a few. In response 
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to question 2, if the outcome of the study was worth the effort, he answered “definitely,” because it gave 
the school a picture of strengths and areas needing increased emphasis. In answer to question 3, how 
the study had benefited or would benefit the school, he said by effecting changes in the writing program 
that would prepare students for success in state testing, their future education, and life. In response to 
question 4, what were suggestions for improving the School Study process, he said yearly reviews [of 
the writing program] and adjustments based on state testing results. 

Seven of the 8 Steering Committee members returned a feedback form. An overview of them 
indicated that some Steering Committee members had duplicated others’ responses: responses from 5 
of the 7 forms were wholly or partly verbatim, 2 even in the same handwriting. 

In response to question 1, what was the respondent ’s personal time spent to conduct the study, 
2 respondents said 20+ hours, 2 said 16 hours, 1 said 30-40 or more, 1 said 25 hours to date, and 1 said 
1 2 hours. In response to question 2, which tasks in the school study required the most time, respondents 
most frequently listed interviews and compiling the report. One respondent, the Steering Committee 
chair, logged many hours at various tasks: interviews — 12 hours, report preparation (including 
compiling interviews) — 20-25 hours, and pulling it all together — 10-12 hours. Also listed by 
respondents were surveys/interviews, the introductory Steering Committee meeting (somewhere 
between 3 and 6 Vi hours), and the rating session (3 hours). 

In response to question 3, how many Steering Committee meetings were held, the Committee 
held 2 Steering Committee planning meetings and a report- writing meeting prior to the faculty rating 
session, then a debriefing meeting following the rating session. 

In response to question 4, whether they could have carried out the study without outside help, 
4 said yes, 2 said no, and 1 said they possibly could have used less help but doubted their ability to do 
it had they had no help at all. To the second part of question 4, what tasks would have been difficult 
without outside help, the 2 respondents who had answered “no” said procedures and getting headed in 
the right direction. 

In answer to question 5, whether the school profile was an accurate picture of their writing 
program 's strengths and weaknesses, 5 said no [2 in capital letters] and 2 said yes. To the second part 
of question 5, what inaccuracies were there and what about the study process may have caused them, 
2 said the [interview] questions did not match the [report] analysis; 1 said the teacher sample was not 
representative since some of the teacher interviewees selected randomly did not truly understand the 
meaning of writing', 1 similarly felt that several teachers did not understand the questions, rendering 
them useless; and 3 did not provide any insights. 

In answer to question 6, if the outcome of the study was worth the effort, 5 said no (4 in capital 
letters) and 2 were undecided at that point pending follow-up and outcomes. One of these latter 2 
respondents was the parent member of the Steering Committee. 
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In answer to question 7, how the study had affected or would affect the school, 5 said “not in 
the least” or something similar, 2 of these respondents saying its only effect was to increase already 
existing confusion and a third saying, “Like everything else, it will soon be forgotten.” One said it 
showed them what was and was not happening in their school related to writing. One respondent, the 
parent, said that there was no effect yet, but hoped that the study showed teachers what areas needed 
work and showed school personnel the need to work together; this respondent added that students 
should always come first, that teachers’ attitudes affect students, and that improved attitudes would help 
the school. 

In answer to question 8, which parts of the study process were most helpful, 4 said none — it was 
a waste of time or useless; and 3 mentioned an aspect of the interviews — illustration of no continuity 
from the top, discrepancies between teacher and student responses (“eye-opening”), or honesty by 
students. 

In answer to question 9, which parts of the process were least helpful, 2 said all, 2 left it blank, 
1 said none, another said none — as long as study outcomes would be implemented eventually, and 1 
said the teacher interviews because the interviewees either misunderstood the questions or were not 
candid in their replies. 

Question 10 solicited suggestions for improvements to the study process. Two suggested more 
time to complete it; 1 suggested more time, training, preparation, and money. Three gave no 
suggestions, 2 stating that they saw no benefit to conducting the study. One respondent, again the 
parent, suggested that the facilitator explain in more detail to the Steering Committee the expectations 
of them and the benefits of the study so that the Committee would not feel it was a waste of time. 

Question 1 1 asked about the utility of the sample report as an aid to report writing. Four 
recommended not including it in the handbook in the future, 2 saying it was misleading. Two found 
it helpful and suggested continuing to include it. One response was ambiguous. 

Reliability of FS2*s report. FS2 submitted a compilation of 8 teacher interviews and a 
compilation of 12 student interviews, as well as 2 administrator interviews. 

All 5 of the report topic responses examined were brief. Three of the 5 responses (to topics 9A, 
9B, and 9E) were entirely or approximately equivalent to the reviewer’s sample in accuracy and breadth 
of data items considered, 2 omitting only a minor data point. The 3 responses incorporated enough 
detail for summarizing and characterizing aspects of the writing program, but they were still concise. 
The report contained less detail overall than the reviewer’s. One of the report responses was partially 
equivalent, having omitted a major data item, and finally, one of the responses was barely equivalent, 
showing much less breadth of data considered and misrepresenting data in the direction of inflated self- 
appraisal. The “Other” space was used in only 1 of the 5 report topics under scrutiny, but was used 
frequently elsewhere in the report for recording additional information. The reviewer’s overall 
assessment of FS2's report, based on the 5 topics considered, is that it was sufficient though abbreviated. 
The assessment of several faculty expressed at the rating meeting was that the report was not 
representative of their writing program. 
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Facilitated School #3 



Description of FS3*s study process. FS3 carried out the self-study in 6 weeks between March 
23 and May 6, 1999, including 1 week of spring break. Five weeks of phone calls and a 5-day 
postponement by the facilitator between February 1 8 and March 23 prefaced the start of the study. The 
facilitator at FS3 supplied much assistance to the school; its study began on her initiative, but was 
eventually organized and completed by the Steering Committee. The facilitator called several times 
before she was able to speak to the principal. The principal then organized the Steering Committee but 
without informing the members. Therefore, at their first planning meeting. Steering Committee 
members did not understand why they were there. The principal later delegated responsibility for the 
study to a staffperson and did not participate in the study process. 

The Steering Committee was composed of 5 members and was headed by the guidance 
counselor and Title I teacher, both of whom assumed responsibility for most of the work in the study 
process. Substitute teachers were hired to cover Steering Committee members’ time out of class. 
Throughout the process, the facilitator initiated all contacts with FS3; the Steering Committee never 
called the facilitator for assistance beyond the services that she offered. However, midway through the 
study process, the principal called a member of the collaborative research team because FS3 had not 
yet received its stipend. 

At the introductory meeting, the facilitator clarified the School Study ’s purpose and benefits 
and then reviewed the steps in the process. In this meeting, she overheard comments such as, ‘“Writing 
detracts from more than enhances the program,”’ and ‘“Some kids can . . .[write], others can’t.’” Also, 
the facilitator gathered that the Committee perceived the principal to be unsupportive of the School 
Study effort and the district to have given few funds to the school for professional development. 

At the next meeting, under the guidance of the facilitator, the Committee selected interviewers 
and interviewees and set dates for the remaining tasks in the process. The facilitator was asked to 
interview the principal and district administrator. The co-chairs of the Steering Committee, the 
guidance counselor and Title I teacher, were chosen to do all the teacher and student interviews. The 
Committee felt that outside interviewers were not qualified to interview, that no parents would, and that 
they did not want the Family Resource Center staff to do it. Because the co-chairs had participated on 
a large number of committees and written several reports during the tenure of the HSE, they felt 
equipped to do the interviewing themselves. Also, they felt responsible for the task since they had been 
appointed by the principal to conduct the study. This planning meeting lasted 4 Yi hours, although the 
facilitator had projected only 1 hour for it. 

The Committee decided to interview more teachers than called for given the school’s size in 
order to include language arts teachers in several grades and other content teachers in accountable 
grades. In a phone interview, one of the co-chairs (and co-interviewers) reported that teacher interviews 
tended to take longer, sometimes miming over a class period by a few minutes. Some interviews appear 
to have been completed as written surveys rather than verbal interviews. Next, the interviewers 
conducted student interviews with groups of 2-3 students, which reportedly worked well even with shy 

I-IO 





147 



students. The student alternates as well as the originally selected interviewees were interviewed; the 
total number of students interviewed is unknown. The co-chair reported that special education students 
were hesitant to answer and needed encouragement. Also, some questions had to be rephrased for lower 
functioning students. Their assessment of the student interviews was that responses from various 
students correlated and that lower functioning students hate writing. All teacher and student interviews 
were spaced over a 3 -week period. The guidance counselor and attendance clerk shared the typing, 
which took approximately 4 days. 

The facilitator interviewed the principal via telephone because of her geographic distance from 
the school. The proposed district administrator interviewee declined an in-person interview with the 
facilitator; instead he submitted responses in writing, which were regarded cynically by the Committee, 
according to the facilitator’s observation. 

The second and third planning meetings of the Steering Committee were attended by only the 
two co-chairs with the facilitator present. The co-chairs also wrote and typed the report themselves, not 
sharing the task with the other Committee members. The facilitator called to check their progress and 
answer questions. The co-chairs spent 3 hours typing the report. 

The purpose of the third planning meeting was to plan the faculty rating meeting. The facilitator 
helped direct them to the specific tasks involved in plarming this meeting. She shared a handout that 
listed the activities of the rating meeting, helped them preassign the faculty small groups, and reviewed 
with them the materials that needed to be ready for the meeting. 

The Steering Committee chairs were still preparing for the rating meeting when the facilitator 
arrived an hour before it began. The facilitator led the rating meeting. One of the co-chairs reported 
that it lasted approximately I'A hours, and that three fourths of faculty members attended. The 
facilitator observed that all participants commented that the responses in the report did not accurately 
reflect their writing program. Also, in a phone interview, one of the co-chairs noted that some report 
topics (i.e., indicators) were incongruous in numbering between the report and the rating form and 
profile. Thus faculty often relied on their own perceptions in addition to the interviews to rate their 
school. However, in some cases they felt compelled to rate according to the data in the report although 
they disagreed with it, one example being the district administrator’s high valuation of his role in the 
writing program. As each small group reported its ratihgs, the facilitator filled in the school profile 
graph. She felt that the resultant school profile was not representative of the school. She observed that 
the ratings ranged from low to high scores in an erratic pattern, some being rated too highly, which she 
somewhat attributed to the confusion caused by the data that contradicted teachers’ personal knowledge. 
In sum, the rating activity presented some complications for the faculty. Nonetheless, according to the 
facilitator, some productive discussion occurred as a result of these discrepancies. Faculty did generally 
arrive at some commonalities of understanding about aspects of the writing program to address, despite 
sometimes having difficulty in reaching consensus. 

The Steering Committee chose to assign the priorities and planning step to the school’s 
Consolidated Planning Committee. In the phone interview aforementioned, which occurred several 
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weeks after FS3's rating meeting, the co-chair stated that she did not know if the priorities step had been 
accomplished yet. 

The facilitator was concerned, given the work and after-school time that the Steering 
Committee members had invested, that they were not confident that they would receive the field-test 
stipend. 

Principal and Steering Committee feedback forms from FS3. The principal completed a 
feedback form. In response to question 1 , whether the writing program profile was an accurate picture 
of FS3's program’s strengths and weaknesses, the principal stated that the school profile gave a 
somewhat accurate picture of FS3's writing program and added that the program is generally weak. The 
principal also said that the faculty have a ‘“higher perspective’” than the students in terms of 
understanding the goals of the writing program. In response to question 2, if the outcome of the study 
was worth the effort, the principal did not yet know, hoping that it would prove productive. In response 
to question 3, how the study had benefited or would benefit the school, the principal said by allowing 
the school to strengthen its writing program and expressed hope that teachers would use the information 
to align writing instruction. In response to question 4, what were suggestions for improving the School 
Study process, the principal suggested that interview questions be aligned to the report form better 
because they did not seem to match. 

Four of the 5 Steering Committee members returned a feedback form. In response to question 
1, what was the respondent’s personal time spent to conduct the study, I respondent said 20+ hours, 
1 said 15 hours, I said 10 hours, and I said 7 hours. In response to question 2, which tasks in the 
School Study required the most time, 2 respondents said the interviews and I said the interviews and 
report. One response was irrelevant to the question. The respondents reported that these activities took 
6-10 hours. 

In response to question 3, how many Steering Committee meetings were held, respondents said 
that 1 planning meeting of the entire Steering Committee was held, in which duties were assigned, 
interviewers and interviewees were selected, and dates set. (However, it is known from phone 
interviews and facilitator data that 2 subsequent planning meetings were convened as well in which only 
the two co-chairs participated.) 

In response to question 4, whether they could have carried out the study without outside help, 
3 said yes and 1 said no. To the second part of question 4, what tasks would have been difficult without 
outside help, 2 respondents said getting started and 1 said asking, compiling, and interpreting interview 
data. 



In answer to question 5, whether the school profile was an accurate picture of their writing 
program ’s strengths and weaknesses, 2 said yes and 2 said no. To the second part of question 5, what 
inaccuracies were there and what about the study process may have caused them, 1 respondent felt that 
the interview data had caused the report to be inaccurate, which generated many differing opinions in 
group discussion at the rating meeting. 
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In answer to question 6, if the outcome of the study was worth the effort, 3 said yes and alluded 
to its potential to improve the writing program. One said no, “because most of the work was done by 
myself and one other person” (this respondent being one of the two co-chairs). 

In answer to question 7, how the study had affected or would affect the school, 2 respondents 
said the information would be incorporated into the writing program by the Consolidated Planning 
Committee; 1 said it had made teachers aware of the importance of different types of writing and how 
different types could be geared to all ages; and 1 said it had increased the entire school’s awareness of 
the importance of the writing program and would align the school toward improving. 

In answer to question 8, which parts of the study process were most helpful, 2 said student 
interviews (e.g., “gave me insight into what students are thinking”), 1 said both teacher and student 
interviews, and 1 said the revelation of the need for professional development and follow-up “to get all 
teachers on the same track.” Also, 1 of the respondents mentioned being unaware that some faculty did 
not know about the teacher’s writing handbook (Writing Portfolio Development Teacher 's Handbook, 
KDE, 1994a). 

In answer to question 9, which parts of the process were least helpful, 2 respondents left it 
blank; 1 said the selected number of teacher interviewees, asserting that most or more teachers should 
have been interviewed to gain a clearer picture; and 1 said that engaging in a review of what was already 
known was least helpful. 

Question 10 solicited suggestions for improvements to the study process. Two suggested 
beginning earlier in the year than April, 1 respondent adding that the study should have been spread 
over a longer time span. One respondent did not have any suggestions. One response was idiosyncratic. 

Question 1 1 asked about the utility of the sample report as an aid to report writing. Three 
respondents said it was helpful and recommended including it in the handbook. One response was 
ambiguous. 

Reliability of FS3's report. FS3 did not submit student interviews, so the reliability 
examination could take into account only the interviews from teachers and administrators. Furthermore, 
since 2 of the 5 topics chosen for review called for the student perspective only, only 3 of the 5 topics 
could be examined reliably. Seven teacher interviews were submitted, but 3 were at least partially 
written in shorthand (and thus illegible to the reviewer), so only 4-5 could be read depending on the 
question. None of the teacher interviews were compiled. 

The examination of the 3 available report topic responses showed that all 3 were entirely or 
approximately equivalent to the reviewer’s sample in accuracy and breadth of data items considered. 
They captured most relevant major and minor points. Two used descriptive detail adequately for 
summarizing and characterizing aspects of the writing program, while 1 had a slight deficit in detail. 
One response reflected a misrepresentation of interview data in the direction of higher self-appraisal. 
Although there were no student interview data available to the reviewer for any of the 5 report topics. 
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the reviewer did read the responses in the student space of the school’s report (for context, see 
Appendix E). Three of the 5 seemed to reflect sufficient breadth in the number of data items considered 
and use of detail, and 2 did not as they were brief, although these impressions cannot be verified. The 
“Other” space was not used in the 5 responses under scrutiny here and was used rarely in the remainder 
of the report. Overall, FS3's report responses were mostly sufficient and descriptive enough for 
depicting the writing program, yet they were sometimes abbreviated. At the rating meeting, the faculty 
expressed the belief that the report was not reflective of the writing program. 

Facilitated School #4 

Description of FS4's study process. FS4 conducted its school study in 5 weeks between April 
19 and May 25, 1999. Ten weeks elapsed between the facilitator’s first phone contact with the school 
on February 8 and the Steering Committee’s start on April 19. The first meeting, originally scheduled 
for March 8, was postponed on late notice at the request of the principal and, due to intervening 
schedule conflicts, could not be rescheduled until April 12. The meeting was again postponed on late 
notice until April 19 due to state testing. 

The facilitator described the principal as very enthusiastic and motivated to do the school study. 
He had read the entire handbook prior to the first Steering Committee meeting. His interest in 
conducting the study emanated from the feeling that the school had experienced a plateau in its writing 
scores. However, the study seemed to commence in a disorganized fashion. For instance, at the first 
meeting, most Steering Committee members arrived without knowing why they were there. Although 
a time line of start-up tasks had been drawn up in one of the facilitator’s phone calls with the principal, 
the involvement of participating individuals was not always requested in a timely fashion. The 
facilitator also reported that the Committee was likewise fairly disorganized; none of the three planning 
meetings had perfect attendance. The study process seemed rushed and disorganized in general. 

There were either 9 or 10 individuals on the Steering Committee (the data sources were 
conflicting), including the principal and at least 1 parent from the School-Based Decision Making 
(SBDM) Council. Almost all faculty on the Committee were language arts teachers. Two members 
were on the Committee because they expressly requested to participate. The first meeting began late, 
after some participants were located (who had not yet been notified of their nomination to the 
Committee) and some copies of the handbook were made. According to the facilitator, most members 
were not anxious to volunteer for tasks, since that time of year was busy. 

At this first meeting, the facilitator first reviewed the handbook. Then the Committee planned 
step 1 . They selected four parents to conduct the teacher and student interviews, and they asked the 
facilitator to train the parents as well as interview the principal and district administrator. The principal 
called the parents and the chair selected the interviewees. A form was sent to parents to gain permission 
to interview students. 

According to the Steering Committee chair, interviews took place over a 1-week period. Five 
teachers and 1 1 students were interviewed. The Committee aimed for teachers’ planning periods and 
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minimal disruptions to students. Several student interviews were conducted individually, while some 
were done in grouns of 2-3 students. The chair reported that students enjoyed the process; she also 
reported that students experienced a problem understanding the jargon of the questions. For example, 
students would discuss examples of real-world writing in their interviews, but when specifically asked 
if they had done real-world writing, would say no. The facilitator reported that the district administrator 
adamantly refused to be interviewed. He requested that she send him the interview form and he 
completed the interview in writing; after reviewing it, the facilitator estimated that he spent little time 
on it. She attempted to follow up by phone to gather more information but was unsuccessful. The 
principal also completed his interview in writing due to an extenuating situation that arose at the school. 
Most interviews were not typed, only a few student interviews by their particular interviewer. 

To write the report. Steering Committee members began working as a group at a meeting 
attended by the facilitator and may have spent extra time individually on their assigned sections, each 
spending a minimum of 3 hours. A parent volunteer or an aide (the data sources were conflicting) 
assembled the various sections and typed the complete report. 

The facilitator had intended to join the Steering Committee at its third meeting for the purpose 
of planning the faculty rating session. She could not due to a conflict, but faxed them detailed 
guidelines for planning, a list of materials needed, and a rating meeting agenda. At this meeting, the 
Steering Committee preassigned small groups for the upcoming rating meeting and decided to ask the 
facilitator to lead the rating session, to which she agreed. 

The Steering Committee chair reported that the faculty rating meeting was slightly longer than 
2 hours and positive, which she attributed to the pre-planning by the committee and preparatory reading 
of the report by faculty. She said that the Committee had projected that they would barely finish rating 
and have no time for discussion. They were pleased that, to the contrary, the meeting progressed 
beyond rating to include completion of the school profile, discussion, and 1 Vi pages of reflections on 
preliminary priorities. She stated that reaching consensus was not an issue. The facilitator also 
described the rating meeting. She said that it “went smoothly, although there was obvious reluctance 
to voice anything . . .negative.” Despite this tendency, the faculty advanced to useful discussion; group 
leaders (Steering Committee members) supported the process and “pushed for honesty,” according to 
the facilitator. She added that some faculty expressed concern that “parents [may not be] the most 
effective interviewers” and also that “they wanted the report to be more reflective of what they saw as 
truth.” 



At the end of the rating meeting, step 4 of the study — priorities and planning — was assigned to 
the Steering Committee and the SBDM Council. The Steering Committee held a fourth meeting for this 
purpose. The facilitator did not attend. 

Principal and Steering Committee feedback forms from FS4. The principal completed a 
feedback form. In response to question 1 , whether the writing program profile was an accurate picture 
of FS4's program ’s strengths and weaknesses, the principal said yes, for the most part. In response to 
the second part of question 1 , what may have caused inaccuracies if any, he said that the vagueness of 
the instrument caused rating inaccuracies on some topics. In response to question 2, if the outcome of 
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the study was worth the effort, he stated that while Committee members may not have realized its worth, 
the faculty’s eyes were opened as discussions took place and plans were made. In response to question 
3, how the study had benefited or would benefit the school, he expected that it would create an increased 
awareness across faculty about the importance of writing instruction. Also, he stated that plans were 
underway to work on teacher training. In response to question 4, what were suggestions for improving 
the study process, he recommended making the paperwork consistent. He suggested that interview 
questions should be matched to report topics for compiling the interview data into the report. Also, he 
said that because the rating form did not contain spaces for rating some of the report topics, they were 
forced to record insufficient information for some topics. 

Seven Steering Committee members (out of either 9 or 10 on the Committee) returned a 
feedback form, including the principal who had also responded to the principal form. His responses to 
questions 5-7 and question 10 (although duplicative or approximate to questions 1-4 on the principal 
form) are recorded with the others below. Also, the third page of the form was missing from all forms, 
inadvertently omitted during copying, so questions 10 and 1 1 were unanswered by these respondents. 

In response to question 1 , what was the respondent ’s personal time spent to conduct the study, 
respondents said 18 hours, IShours, aminimumof 12, 10 */ 2 , 6, 3, and 2 hours. In response to question 
2, which tasks in the School Study required the most time, respondents gave multiple answers. Five of 
the 7 respondents said compiling the interview data into the report, 2 of them estimating a minimum 
of 3 hours spent. Four of the 7 listed meetings. These were the two frequent responses. Isolated 
mentions from different individuals were copying, discussing the report, organizing, and acting as 
school liaison. 

In response to question 3, how many Steering Committee meetings were held, the Committee 
held 4 planning meetings, including one for report planning and report writing and one following the 
rating session for beginning step 4. However, all members were not present at each meeting, 
individuals’ attendance ranging between zero to four meetings. 

In response to question 4, whether they could have carried out the study without outside help, 
2 said yes (1 conceding, however, that the study may have been too great a task otherwise). Two said 
no. Two answered ambiguously and 1 left it blank. To the second part of question 4, what tasks would 
have been difficult without outside help, respondents listed providing motivation to do the study, 
imderstanding the process in general, interviewing administrators, and leading the Committee and 
faculty meetings. One respondent referred to the help of a volimteer who typed the report and to the 
help provided for conducting all of the interviews. 

In answer to question 5, whether the school profile was an accurate picture of their writing 
program ’s strengths and weaknesses, 3 said yes, 2 said mostly, 1 said no, and 1 answered ambiguously. 
To the second part of question 5, what inaccuracies were there and what about the study process may 
have caused them, all 7 respondents gave either one or two responses related to interviews and report 
compilation. Four felt that interviews were not representative — student interviews were not valid, the 
student sample was not large enough, or interviewers did not imderstand the writing program enough 
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to gather accurate information. Five said that interview questions were either too vague or did not 
correlate well to some report topics, making assimilation into the report insufficient or difficult. 

In answer to question 6, if the outcome of the study was worth the effort, 5 said yes; 1 did not 
know (i.e.,“time will tell”); and 1 did not answer directly, stating that it was difficult for those on the 
Committee who were already burdened with writing instruction duties and suggesting that the inclusion 
of more content area teachers outside language arts on the Committee would have helped. 

In answer to question 7, how the study had affected or would affect the school, 2 referred 
generally to improvement in the writing program. Three of the 7 referred to specific improvements, 
described as follows. One respondent described a chain of events in which new direction would 
translate into the development of a writing program instructional policy by the SBDM Council, which 
would in turn “eventually [permeate] change into the writing program.” Another respondent also 
described a series of events in which school-wide teacher awareness would “[cause] writing instruction 
[to] be a goal of every content teacher . . .[and] foster more types of writing school-wide.” A fifth 
respondent said benefits to professional development and SBDM Council decisions would result. 
Finally, 1 respondent predicted little effect based on past experience and 1 respondent answered 
ambiguously. 

In answer to question 8, which parts of the study process were most helpful, 4 of the 7 
respondents gave multiple answers. Five of 7 listed one or more aspects of the rating meeting — faculty 
discussion, the school profile and the process of developing it, and/or goal-setting. Two mentioned 
discussion and work done specifically within the Steering Committee. Two said the report, 1 said the 
interviews/surveys, and 1 said the student interviews specifically. 

In answer to question 9, which parts of the process were least helpful, 4 pointed to the design 
of the handbook or to peoples’ use of it. Specifically, one respondent mentioned vague interview 
questions and lack of their alignment to report topics for report compilation. One mentioned the school 
profile due to mistakes in the instrument. A third respondent said the ambiguity and inconsistency of 
interview responses, which caused doubt about the validity of the questions. A fourth said the room 
for interpretation due to the interview/report responses not being representative. Three of the 7 
respondents said no parts were least helpful or had no opinion. 

As explained earlier, questions 10 and 1 1 were missing from all of the feedback forms due to 
an inadvertent omission of that page during copying. 

Reliability of FS4's report. FS4 submitted 5 teacher interviews and 1 1 student inter\'iews, as 
well as 2 administrator interviews. Teacher interviews were not compiled. Nor were individual student 
interviews compiled. However, responses of 5 students who were interviewed in two separate groups 
were recorded on two forms. Thus, two forms containing responses of 5 students, plus 6 individual 
student interviews, were submitted. 

Four of the 5 report responses were entirely or approximately equivalent to the reviewer’s 
sample in accuracy and breadth of data items considered. One response contained a slight omission and 
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1 response a slight misrepresentation of the interview data. While the fifth report topic response 
contained a substantial misrepresentation of the interview data, it was nevertheless deemed 
approximately equivalent to the reviewer’s sample based on the breadth and quality articulation of the 
other data considered. The “Other” space was not used at all throughout the report, probably because 
it was unnecessary given the exceptional amount of commentary that was already included in the 
designated role group spaces. In sum, all 5 topics were comprehensive and showed a deft use of detail, 
incorporated skillfully for summarizing and characterizing aspects of the writing program. In contrast, 
at the rating meeting, several faculty expressed that the report was not as reflective of the writing 
program as it could have been because some of the interviews were not. 

The facilitator offered some insights into the level of representativeness achieved in the 
interviews. For several years, the school has received little district support, which has transferred much 
additional work to the school faculty. Her interpretation of the interview responses was that both 
interviewees and interviewers skirted around this controversial issue rather than overtly state it. For 
instance, to record administrator statements, they sometimes used quotation marks to punctuate them 
as if to cast doubt on their validity. The facilitator empathized with their caution as she found herself 
being careful as well; she raised the point that it is probably unnecessary to put controversial statements 
in writing when the rest of the faculty already understand the situation dynamics. 

Facilitated School #5 

Description of FSS's study process. FS5 conducted its school study in 7 weeks during a 9- 
week period between March 29 and June 1, 1999 (the study was suspended for a week each of spring 
break and state testing). Four weeks of phone calls between the facilitator and the school predated a 
February 22 meeting between the facilitator and assistant principal only. Another 414 weeks transpired 
before the first full Steering Committee meeting on March 29. 

The facilitator reported that the Committee’s approach to and progress in the study was 
wonderful. The assistant principal acted as the Steering Committee chair. The facilitator described her 
as a very energetic and strong leader. The chair needed a small amount of help to start, i.e., “tell me the 
quickest and easiest way of doing this.” She had adroitly prepared the Steering Committee so that they 
were ready to begin when the facilitator first met with them. The Committee chair phoned and e-mailed 
the facilitator fairly often and requested her presence at meetings. The principal of FS5 also involved 
himself in the study. 

The Steering Committee was composed of 4 members, including the assistant principal as its 
chair. At its first meeting, attended by the facilitator, the chair presented the School Study handbook 
to the rest of the Committee. She presented the study process as a compliment to the school for having 
been asked to participate, as an opportunity to evaluate the writing program, and as a professional 
development activity that would enhance the faculty’s growth. The Committee reviewed the four steps 
of the study in-depth and began planning the interviewing step. The chair reported that it was helpful 
having a few copies of the handbook at the meeting. She found the handbook to be helpful and orderly 
in its organization. The facilitator was also present at a second meeting, at which the Committee 
finalized the selection of interviewees, the assignment of report sections, and the schedule for the 
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remainder of the study. Following the second meeting, the facilitator assessed the school’s motivation 
positively, “All the teachers feel that this is going to benefit them greatly. I have enjoyed all my time 
with them. These folks are hard workers.” 

The Steering Committee chair reported that it was easy to select the interviewers — two substitute 
instructional assistants who interviewed as a pair. She stated that they dreaded the task initially but 
were engaged by it once they began. They interviewed 7 teachers and 10 students. They chose to 
interview students individually instead of in groups because they believed that students would give 
better responses individually. It appeared that the interviews of the older students were in their 
handwriting, indicating that these students either responded via written survey or were permitted to 
write their responses in the course of being verbally interviewed; it is unknovsm which is the case. 
Interviewers were able to stay within a class period for student interviews. Teacher interviews took a 
long time at first, but became easier for the interviewers with practice; some of these also seem to have 
been administered via written survey. The principal was interviewed by one of the two interviewers; 
it is unclear whether one also interviewed the district administrator or that individual completed the 
interview in writing. All interviews were accomplished in 2 days and were not typed. 

The Committee worked collaboratively to write the report, spending 4 hours per day for 3 days. 
The chair mused that writing the report together afforded the opportunity for discussing what positive 
and negative data to accentuate. The report was typed by an individual designated as the word 
processor, not a member of the Steering Committee. 

Until she was asked about the rating meeting in a phone interview, the Steering Committee chair 
had forgotten about the subsequent steps in the study — the rating meeting and setting, 
priorities — thinking that they had finished the process with the report-writing activity. Once reminded, 
she scheduled the rating meeting and described it in a later phone interview. She commented that the 
rating meeting was “the biggest ordeal of the whole process.” So that all faculty would be familiar with 
all report topics, the Committee decided to conduct the rating session as a whole group rather than 
divide the faculty into small groups. The meeting lasted 2 hours, which the chair felt was too long. 
Since faculty were unfamiliar with the report categories, they had to spend time discussing them. The 
faculty reached consensus on everything, according to the chair. They ranked low on some issues out 
of their control such as community support and district office support, so they were unsure how to 
address them. However, they identified instructional issues that they felt they could address and 
improve. The group also expressed the need for a clearer student perspective, thinking that for the size 
of the student body, 1 0 interviewees were perhaps too few. 

To set priorities, the Committee planned another faculty- wide meeting for the last day of school, 
saying that they wanted “to share everything with everyone.” The chair called the research team mid- 
summer to inquire about the stipend as they had not yet received it. In the course of this conversation, 
she reported that they had cancelled the priorities meeting, thinking that it would be more beneficial to 
hold it at the beginning of the new school year when faculty would be fresh. However, the school 
profile had been presented at a regular faculty meeting before the close of the year. Faculty were asked 
to be thinking about the results for the next school year. The faculty would return in early August to 
attend 2 days of professional development on curriculum issues to be given by consultants. The 
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Committee’s hope was to include writing instruction in these discussions; their intention was that 
responsibility for writing instruction not be “dumped” on the writing teacher but holistically 
incorporated into all instruction. Earlier in May, the facilitator had shared her view that FS5 would 
progress far with the School Study, they were asking questions such as, “What can we say the last day 
of school?” and “What will this lead to next year?” 

Principal and Steering Committee feedback forms from FS5. The principal completed a 
feedback form. In response to question 1 , whether the writing program profile was an accurate picture 
of FS5's program ’s strengths and weaknesses, he responded affirmatively. In response to question 2, 
if the outcome of the study was worth the effort, he said yes but suggested that for the time spent, all 
faculty be interviewed in future studies. In response to question 3, how the study would or had 
benefited the school, he stated that any feedback is helpful for designing and improving instruction that 
increases student achievement. In response to question 4, what were suggestions for improving the 
School Study, his suggestion was unintelligible. 

Four individuals returned a feedback form, 2 of whom did not appear to be original members 
of the Committee — the interviewer and the person who did the word processing; yet, the latter indicated 
that she did perform some Steering Committee tasks in addition to word processing. 

In response to question \ ,what was the respondent ’s personal time spent to conduct this study, 
1 respondent said 22-24 hours, 1 said 21 hours, 1 said 10-12, and 1 said 6. In response to question 2, 
which tasks in the School Study required the most time, 1 said interviews preparation (4-6 hours); 1 said 
copying interview forms and assembling packages for the interviews (2-3 hours) and analyzing 
interviews (2 hours); and 1 said reading each individual reply in the interviews. 

In response to question 3, how many Steering Committee meetings were held, the Steering 
Committee held either 3 or 4 meetings prior to the faculty rating session. (The responses are 
conflicting; it is possible that not everyone attended all meetings.) 

In response to question 4, whether they could have carried out the study without outside help, 
3 either indicated or answered yes, 2 of them saying, “most of the support was contained within the 
Committee.” One did not respond. To the second part of question 4, what tasks would have been 
difficult without outside help, 2 suggested either that an additional interviewer or only non-staff 
interviewer(s) be employed, given the size and time constraints of faculty, and 1 said that the most 
difficult task was finding the time to gather needed information. One of these 3 added that answering 
a survey in the role of interviewee also took a substantial amount of time. 

In answer to question 5, whether the school profile was an accurate picture of their program ’s 
strengths and weaknesses, 3 said yes and 1 did not answer directly. To the second part of question 5, 
what inaccuracies were there and what about the study process may have caused them, the latter 
respondent mentioned above identified the student interviews as a source of inaccuracies, as student 
interviewees may have forgotten writing concepts previously taught to them. The other 3 did not list 
anything. 
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In answer to question 6, if the outcome of the study was worth the effort, 3 said or indicated yes, 
i.e., “any feedback . . .is welcomed and beneficial.” A fourth respondent said somewhat and added that 
the study should have been initiated and completed earlier in the school year (before May). 

In answer to question 7, how the study had affected or would affect the school, 2 said by 
identifying areas needing improvement and then causing those improvements to be made. One said it 
had increased faculty’s awareness of their knowledge level and participation in the writing program. 
One was not sure. 

In answer to question 8, which parts of the study process were most helpful, 2 said the teacher 
interviews, either because teachers demonstrated honesty or because they provided the most information 
on writing instruction. One gave 3 responses: (a) the student and teacher interviews because they are 
an accurate portrayal of what is actually taking place in the classroom; (b) recording the interviews (in 
the report); and (c) the administrators’ interviews, because principals help with professional 
development and realigning curriculum and schedules. A fourth pointed to his or her answer to 
question 5, which stated that only student interviews showed inaccuracies, due to students forgetting 
what had been taught to them; it is unclear whether the respondent meant that student interviews 
provided insights because of their inaccuracies or that all other parts of the study but the student 
interviews were helpful. 

In answer to question 9, which parts of the study were least helpful, 1 said the randomness of 
student interview selection and the tedium and unclear directions for the rating process. One said the 
lack of supportive detail in student responses to interview questions. One said having to ask some of 
the students about benchmarks, because the multi-part questions frustrated them. One said none. 



Question 1 0 solicited suggestions for improvements to the study process. One said to condense 
the number of interview questions. One said to gain input from a greater number of teachers to have 
a greater impact. One had five suggestions: (a) conduct the study earlier in the school year; (b) allow 
teachers time to preview interview questions to speed up the interview process; (c) allow teachers to 
be involved in selecting the students to be interviewed; (d) include all teachers as interviewees; and (e) 
condense the number of interview questions or allow teachers to write answers to most questions prior 
to their interviews. One had no suggestions. 

Question 1 1 asked about the utility of the sample report as an aid to report writing. Three said 
it was useful and recommended continuing to include it in the handbook, and 1 had not used it. 

Reliability of FSS's report. FS5 submitted 7 teacher interviews and 1 1 student interviews. 
Neither set was compiled. Also, FS5 submitted a principal, but not a district administrator, interview. 

A few of the 5 report topic responses examined were rather brief in at least part of the response. 
Two of the 5 responses were entirely or approximately equivalent to the reviewer’ s sample in accuracy 
and breadth of data items considered. These 2 responses accounted for major and minor relevant points 
and effectively incorporated detail for summarizing and describing aspects of the writing program; 
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however, 1 of the responses did contain a slight misrepresentation of the interview data. Two other of 
the 5 responses were partially equivalent to the reviewer’s sample in accuracy and scope; they omitted 
a few major points and lacked enough supporting detail for describing aspects of the writing program. 
Finally, 1 of the 5 responses was barely equivalent to the reviewer’s sample in accuracy and scope, 
having omitted all major points, one minor point, and much detail; however, the minor point that was 
included in the response was elucidated well. The “Other” space was not utilized at all throughout the 
report; however, the review of the 5 responses under scrutiny here showed that additional information 
was incorporated directly into the responses of the designated role groups. In sum, FS5's report was 
concise and only partially comprehensive. 

Condition 2: Independent Schools 

Independent School #1 

Description of ISl’s study process. ISl conducted the School Study in 6 weeks over an 
8-week period between March 8 and May 5, 1 999, suspending the study for a week each of spring break 
and testing. Beginning with a phone call on February 2 from the principal to the contact person, 5 
weeks elapsed before the Steering Committee first met sometime during the week of March 8. The 
principal chose the second week of March to start the study so that it could be completed before testing 
began in May. The contact person said that the principal had called her once to inquire about the receipt 
of the stipend and to ask two questions about the study process: whether it was acceptable to use parents 
rather than teachers as interviewers and what was the recommended size of the Steering Committee. 
The remainder of the phone contacts were initiated by the contact person; she checked with the principal 
each month for an update on IS I's progress. Otherwise, the contact person reported that the principal’s 
initial impression of the School Study handbook was that it was ‘“easy . . . [for accessing] information, 
well put-together, and divided nicely.’” Near the conclusion of the study, the principal thanked her, 
saying that the study was “‘a great process, [providing] very useful information.’” 

The principal provided a synopsis of the first Steering Committee meeting in separate phone 
conversations with the contact person and another member of the collaborative research team. There 
were 4 on the Steering Committee — 3 teachers and the principal, who served as the chair. The 
principal presented the School Study handbook, having already made a copy for each Committee 
member. He overviewed the process, then discussed each of the four steps in more detail. The 
Committee then selected interviewers and student and teacher interviewees, and set the dates for the 
interviews. In addition, having read the teacher interview questions, the Steering Committee began 
brainstorming about ways to gain whole faculty buy-in so that corrective changes could be made to the 
writing program. A parent who was also an educator was chosen as the interviewer; the Committee 
chose her because staff knew her and they believed staff would be comfortable with her. 

For the interviews, the interviewer brought a partner to help scribe for her. They returned to the 
school several times over a period of approximately 10 days to complete interviews, twice for teacher 
interviews and thrice for student interviews, which added up to 2-3 days total. They interviewed 6 
teachers and an unknown number of students. (It is unknown how many students were interviewed at 
this school because the school did not submit copies of its interviews to the research team and the 
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principal did not say during a phone interview.) For the most part, teacher and student interviews could 
each be conducted within the space of a class period, although some tended to run over the allotted time. 
Initially, the interviewers talked to students on an individual basis, but switched to interviewing them 
in groups of 2-3 because of time constraints. According to feedback they gave the principal, the 
interviewers preferred the group method. The interviewer typed onto the computer while the scribe 
wrote in longhand during the interviews, thus no typing was done afterwards. The scribe’s notes were 
to serve as a cross-check of the interviewer’s notes. 

The principal was unaware of the amount of time spent by each Steering Committee member 
in developing sections of the report or the total time spent writing and typing it, but said that each 
Steering Committee member reported that the task required a lot of time. 

The principal reported that the faculty-wide rating meeting included the rating activity, school 
profile, and action-plan activities. It lasted 3 hours, but the rating itself took just 45-60 minutes. They 
did the rating activity in small groups to which they were preassigned by the principal. Regarding step 
4, the principal reported that they did not finish the action plan but said that they have a good handle 
on a focus for next year. He told the contact person that they were planning to focus on professional 
development in writing for all faculty since learning that, currently, writing was being targeted only in 
accountable grades. 

Principal and Steering Committee feedback forms from ISl. The principal provided brief 
responses on the feedback form. In response to question 1 , whether the writing program profile was 
an accurate picture of ISl 's program 's strengths and weaknesses, he responded affirmatively. In 
response to question 2, if the outcome of the study was worth the effort, he said yes. In response to 
question 3, how the study had benefited or would benefit the school, he answered that they had targeted 
areas for professional development for all staff members. In response to question 4, what were 
suggestions for improving the School Study, he had none. 

Three of the 4 Steering Committee members returned a feedback form. (Presumably, the 
principal did not complete one because he had responded on the designated principal form.) In response 
to question 1, what was the respondent 's personal time spent to conduct the study, 1 spent IVi hours, 
1 spent 8-9 hours, and 1 spent 12 hours. In response to question 2, which tasks in the School Study 
required the most time, 1 said reading the manual (3 hours) and preparing the report (2 hrs); 1 said 
reading, studying, and doing the report; and the third said reading the interview responses and trying 
to correlate the questions and responses to report topics. So, all 3 respondents mentioned the task of 
creating the report. 

In response to question 3, how many Steering Committee meetings were held, the Committee 
held 4 plaiming meetings prior to the faculty-wide rating meeting. 

In response to question 4, at what points in the study did they need to contact someone with 
questions, 1 said she had a few questions but did not specify; 1 said she discussed the Writing Portfolio 
Development Teacher 's Handbook (KDE, 1994a) with another teacher; 1 said she found everything to 
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be fairly straightforward. To the second part of question 4, for which tasks did they need more 
assistance than was available, none responded. 

In answer to question 5, whether the school profile was an accurate picture of their writing 
program ‘s strengths and weaknesses, 1 said “basically a true picture.” One answered yes, and added 
that she expected additional strengths and weaknesses to surface later. 

In answer to question 6, if the outcome of the study was worth the effort, 1 said “Yes! Yes!”; 1 
said yes, mainly by making nonaccountable grade teachers aware of the difficulty in improving writing 
without their participation; and 1 said no, because the amount of time spent interviewing teachers 
confirmed what was already known and the sample of student interviews was not thorough enough. 

In answer to question 7, how the study had affected or would affect the school, 1 said by creating 
awareness by all teachers of writing impact for all students; 1 said by creating awareness of areas 
needing development and by stimulating new work on expanding the curriculum to overcome those 
weaknesses; and 1 said by highlighting the need for professional development in writing instruction for 
all teachers. But the third respondent doubted the teachers’ willingness to “take ownership of the ‘next 
steps’ in the action plan,” saying that the teachers thought of the rating session as just “another thing 
to do.” 



In answer to question 8, which parts of the study process were most helpful, 1 said the student 
and teacher interviews; 1 said seeing “how the students perceive their writing and the actuality of it”; 
and 1 said the rating guide. The third respondent added, however, that only the Steering Committee 
saw the rating guide; she wished that the entire faculty had seen it “to provide a vision of what’s 
possible . . .our faculty meeting did not provide a vision— just numbers.” 

In answer to question 9, which parts of the study process were least helpful, 2 said all parts were 
helpful, but 1 added that some report topics could not be answered from the interview data. One left 
it blank. 

Question 1 0 solicited suggestions for improvements to the study process. Two respondents were 
unclear. One respondent had two suggestions. First, the respondent suggested aligning the report topics 
to specific questions in the interviews because teachers are busy and could use that help. Second, the 
respondent presented a recommendation for greatly expanding the scope of the rating session. This 
individual stated that “a faculty meeting . . .‘centered on’ goals and steps for improvement needs more 
than numbers and printed pages.” Instead, the rating meeting should have been a workshop in which 
teachers wrote down specific actions to be taken and left with “specific steps in mind,” rather than just 
“a brainstorm list on a chart.” In addition, the respondent stated that such a workshop should model 
writing: “I figure that if we want kids to write, then we teachers need to write arid it needs to be 
modeled in the discussion and decision-making process.” 

Question 1 1 asked about the utility of the sample report as an aid to report writing. One used 
it but said it was not necessary. One said yes, she had used it. One did not see it. 
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Reliability of ISl 's report. IS 1 provided a copy of its report but was not comfortable providing 
the interviews. Therefore, the report could not be truly examined for reliability. However, a review 
of the report revealed it to be abbreviated with very little detail, capturing some points but not others 
and presenting only one-sentence statements under a few role group spaces. Also, the contact person 
shared her assessment of the representativeness of the report. Based on anecdotal feedback that she had 
received from the interviewers about the interviews, she believed that the writers were diplomatic in 
the report and that they did not incorporate all data. She felt that perhaps they w'cre unwilling to say 
that they were dissatisfied with their progress in the writing program. 

Independent School #2 

Description of IS2's study process. IS2 conducted the School Study in 6 weeks between March 
1 1 and April 21, 1999. The contact person first called the school principal sometime in late January 
to introduce herself and offer assistance as necessary. She stated that he was excited about the study 
and what the school could leam from it. She called periodically throughout the study to inquire if he 
or the Steering Committee had any questions. In a late March call when the interviews were almost 
completed, the principal stated that they had no questions or need of assistance; the interviews were 
going well. He continued to be delighted with the process, although the timing was poor because the 
school had experienced problems that year and state testing was taking place. During a later call from 
the contact person, about a month after the rating meeting, he had a few questions about the action plan. 
Throughout the study, the contact person initiated all contacts except for one call and one e-mail 
communication, both from the Steering Committee chair. 

The district writing cluster leader served as the Steering Committee chair. In a phone interview, 
she also said that the timing of the study was poor due to contemporaneous portfolio scoring. Also, 
teachers at the feeder school declined to help with the study, putting the burden on the teachers at IS2. 
According to the chair, there were 5 or 6 on the Steering Committee (data sources were conflicting), 
including herself and various faculty members. Except for one teacher, nonaccountable grade teachers 
served on the committee to free accountable grade teachers for portfolio scoring. 

The Steering Committee chair organized most of the study for the other members due to their 
limited time. The committee had only 1 main plaiming meeting to start the study, at which the chair 
presented the School Study and set dates, and another short one to preassign teachers for the rating 
session. 

The chair was chosen to be the interviewer since she had no classes. She also selected all of the 
interviewees. IS2 included two grades from its feeder school in the study, interviewing both teachers 
and students. She surveyed 7 teachers and interviewed 1 1 students. To save time, teachers were asked 
to respond via written survey rather than verbal interview to save time. The chair interviewed students 
in groups of 4 because she believed that they would be most comfortable in that group size; she reported 
that, as expected, this group size worked well because students were openly expressive. In one 
particular group in which the interviewer knew none of the students, she was “convinced” that they 
would not have been forthcoming in responding had the interview been conducted on an individual , 
basis instead. Student interviews lasted approximately 45 minutes; the interviewer spent 2 days total 
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on the task, 1 day at each of the schools. She scribed student interviews herself by taking notes and 
spent 1 hour typing and compiling them onto one student form. 

To complete the report, the Steering Committee chair said that most Steering Committee 
members spent approximately VA hours on their assigned sections. The chair’s secretary typed the 
report over the week of spring break. 

At the faculty-wide rating meeting, language arts teachers (some from the Steering Committee 
and some not) led the small groups. Each group completed its portion of the ratings, but groups did not 
share their findings at this point. The Steering Committee had decided to assign the first part of step 
4, setting priorities, to the whole faculty. Thus another faculty-wide meeting was convened for this 
purpose. The committee divided groups further for this meeting. Ratings were shared on the school 
profile at this time. Each of these two faculty meetings lasted about 45-60 minutes. 

The second part of step 4, planning actions, was assigned to a planning committee composed 
of the Steering Committee participants and a few additional individuals. The planning committee 
intended to complete planning during the summer before the new principal was scheduled to arrive and 
gather suggestions from the feeder school the following school year. 

Following IS2's completion of the school study, the Steering Committee shared some insights 
about their progress and findings. The study confirmed what they knew — that there had previously 
been no buy-in to the writing program from non-language arts teachers. The school had experienced 
a bad year because some non-language arts, content teachers did not view writing instruction as their 
job. Several did not want to be involved in the school study. IS2 was hoping for a better school year 
in 1999-2000. The chair was pleased with the process and glad to be a part of it. She had particularly 
found the experience of interviewing students to be interesting and valuable. By May, IS2 was 
considering next steps and had already incorporated some information from the student interviews into 
the Consolidated Plan. 

Principal and Steering Committee feedback forms from IS2. The principal provided 
extensive comments on the feedback form. In response to question 1, whether the writing program 
profile was an accurate picture of IS2's program ’s strengths and weaknesses, he believed that the 
school profile very accurately depicted IS2's writing program, from which an improvement plan could 
be developed. In response to question 2, if the outcome of the study was worth the effort, he felt that 
the study was “well worth the effort,” although the timing was not the best, and recommended that 
every school participate in the study. 

In response to question 3, how the study had benefited or would benefit the school, he had 
several insights. He stated that the study permitted the school to move beyond perceptions, 
demonstrating areas in which they could improve student writing in nonaccountable as well as 
accountable grades. The results enabled the administrators and language arts teachers to develop a plan 
that would involve all teachers, because most non-language arts teachers still lacked a comprehensive 
knowledge of the requisite skills to complement language arts teachers’ instruction. Specifically, they 
were planning changes to the Title I program in order to provide more training and professional 
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development. The SBDM council approved hiring a writing specialist to assist in the development of 
the school’s writing plan, to train teachers, and to work closely with accountable grade teams on 
developing students’ writing portfolios. 

In response to question 4, what were suggestions for improving the School Study, the principal 
alluded again to its timing. He was not sure that there is a best time to conduct the study. It was a 
“stretch” for the school to complete it in the spring; he added, however, that it was probably the 
“teachable moment” since that was the time when portfolios were on the mind of accountable grade 
teachers who were in the midst of scoring them. 

Three of the 5 or 6 Steering Committee members, including the chair, returned a feedback form. 
In response to question 1, what was the respondent ’s personal time spent to conduct the study, 1 said 
10 hours, 1 said VA to 4 hours, and the chair said 5 hours (although the chair’s estimate seems low 
given the tasks that she performed and the time that she reported on the phone having spent on those 
tasks). In response to question 2, which tasks in the School Study required the most time, 1 said 
organizing interviews at both schools (1 hour) and filling out the report based on all sources of 
information (3 hours); 1 likewise said gathering and consolidating the interview data hours); and 
the third also said reviewing the questionnaires and summarizing the information. Thus each identified 
the process of translating interview data into the report as the task requiring the most time. 

In response to question 3, how many Steering Committee meetings were held, the Steering 
Committee held 2 planning meetings prior to the faculty meetings. However, 1 of the 3 respondents 
had attended none. 

In response to question 4, at what point did they need to contact someone with questions and 
for which tasks did they need more assistance than was available, 2 said none, 1 adding that all was 
explained and all her/his questions answered at the initial meeting. The chair said that only the final 
report was an issue (it is unclear whether she actually meant the report or the action plan, at which point 
it is known that the principal did call with questions). 

In answer to question 5, whether the school profile was an accurate picture of their writing 
program ’s strengths and weaknesses, 2 respondents said yes and 1 left it blank. One added that in fact, 
the study produced similar results to another study of IS2's writing program conducted earlier, which 
also showed the existence of definite needs of the program. 

In answer to question 6, if the outcome of the study was worth the effort, all 3 respondents said 
yes, 1 adding, “. . .although the time line was ridiculous — too much work in too . . . [small] a time 
frame.” 



In answer to question 7, how the study had affected or would affect the school, 1 said it raised 
awareness of how each role group views the school’s writing process. One listed two effects: (a) the 
creation of a committee to study various options for addressing weaknesses and (b) the hiring of a 
writing teacher solely for working with teachers and addressing concerns expressed by faculty and 
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students. One believed, admittedly sadly, that the school would not be affected because the 
“uncovered” information was already known and apathy was their biggest obstacle. 

In answer to question 8, which parts of the study process were most helpful, all 3 said the 
interviews, 1 stating that the interviews illustrated different perspectives but common concerns. 

In answer to question 9, which parts of the process were least helpful, 2 said the time frame and 
I said none because all parts were necessary to achieve the final results. 

Question 10 solicited suggestions for improvements to the study process. Two suggested 
aligning the interviews with the report. One gave none. 

Question 1 1 asked about the utility of the sample report as an aid to report writing. All 3 found 
it helpful. The chair also noted that Steering Committee members had indicated to her that the sample 
was helpful. 

Reliability of IS2*s report. IS2 had intended to send its report along with the interviews but 
neglected to include it; therefore, a reliability examination could not be performed. The contact person 
could offer no insight because she did not see the school’s report. 

Independent School #3 

. Description of IS3*s study process. IS3 did not complete the study of its writing program. The 
contact person began calling the school in January to find out if they had decided to conduct the study. 
In a February call, she was told that responsibility for it had been given to the assistant principal. The 
assistant principal communicated at that time that she understood the process and had no problems. 
When the contact person called again, a member of the SBDM council told her that IS3 was preparing 
to hold a faculty meeting to glean faculty’s wishes about conducting the study. When she called again, 
she was told that the decision had been made to proceed and dates were set for steps in the study. She 
then made another call to confirm the school’s date for the faculty rating meeting, which she was told 
would be April 5, 1999. The contact person initiated all phone calls and contacts with the school. 

In mid-May, the contact person spoke to the assistant principal, who informed her that the 
interviews had not yet been done. The contact person offered to help the Committee with the 
interviews. Later, she was told that teacher interviews were completed and student interviews would 
be conducted by the assistant principal during the last 2 days of school. Also, the assistant principal 
indicated that the Steering Committee would vmte the report the day after the close of school. They 
would hold the rating meeting during summer school around mid-June with only a few faculty present. 
The contact person offered to help with the meeting and suggested that the school pay faculty to 
participate. 

The assistant principal explained the various reasons for IS3's delay in doing the study. The 
SBDM council had withdrawn support for the study midway through teacher interviews, saying that 
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the study was taking too much time. The study was suspended until the principal overrode the decision. 
Also, a Steering Committee member had extenuating personal health concerns. 

Subsequent contacts and offers of assistance by the contact person were not acknowledged by 
the school. Based on a research team member’s conversation with the Steering Committee chair in 
early July, it was gathered that no steps beyond interviewing had been completed. The school had no 
data to contribute to the field test. 

Principal and Steering Committee feedback forms from IS3. No participants from this 
school returned a feedback form. 

Reliability of IS3's report. Although IS3 reported that it had completed the interviews step, 
this cannot be confirmed. No documents were received from IS3, so a reliability examination could 
not be performed. 

Independent School #4 

Description of IS4's study process. IS4 completed its school study in 6 weeks between early 
March and May, 1999. The study was suspended for 2 weeks while assessment and portfolio scoring 
were taking place. The contact person made several phone calls prior to and in the beginning of the 
study, always speaking to the principal. The principal assured her that they were handling the process 
ably and that the Steering Committee chair was very efficient. He called the contact person twice with 
minor questions. After identifying individuals’ roles early in the study process, IS4 postponed the study 
until assessment was over. The contact person had not spoken to the principal since that time as a few 
of her calls had not been returned, but was confident in IS4's ability to conduct the study because of the 
principal’s efficiency. The principal had, however, spoken to another member of the collaborative 
research team in the interim and given a tentative date for the school’s rating meeting. 

In a phone interview, the Steering Committee chair relayed that the responsibility came to her 
late because the writing leader teacher originally intended for the position ultimately could not fill it. 
The study began late because of this circumstance as well as the delay due to testing and portfolio 
scoring. There were originally at least 4 on the Steering Committee (the data were incomplete); 
however, the parent member could never be reached at. an opportune moment. Thus the Committee 
eventually had 3 members, including the chair. Also, the two other teacher members were only 
sporadically available due to various personal reasons. Thus the chair did a substantial portion of the 
work herself 

The Steering Committee had 1 planning meeting; other exchanges between members were 
informal. At the meeting, the chair presented the School Study handbook to the Committee, gave each 
member a copy, and assigned report sections. She also did the principal interview, copied interviews 
for the Steering Committee, wrote a section of the report, typed the entire report, and copied it. 

Seven teachers and 12 students were interviewed. The principal made the decision that most 
teachers should be interviewed. Two staff from a local Family Resource Center interviewed teachers 
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over a 1-week period; three higher grade level students from another school interviewed 4 students each 
over a period of 4 days within the same week. Students were interviewed on an individual basis; the 
chair said that students were not sufficient interviewers because they did not probe student interviewees 
for more comprehensive, meaningful responses. Also, the chair reported that some student interviewees 
did not understand the questions. Interviews lasted about 45 minutes each. As planned, the chair 
interviewed the principal, but she administered the district administrator interview to the superintendent 
via written survey because she herself became ill on the scheduled date. No interviews were typed. 

The report writing duties were divided among 3 Steering Committee members. The chair was 
unsure how long it took the others to do their sections; it took her 2-3 hours to complete her section and 
1 hour to type it. 

The faculty rating meeting was rescheduled several times . The faculty’ s presence was requested 
on a voluntary basis because the meeting was held so close to the end of the school year. Slightly more 
than 50% of the faculty attended. Although she desired higher attendance, the chair noted that the 
number attending was adequate for the task because the faculty is large. The chair reported that the 
meeting progressed well and did not last as long as was expected (it lasted VA -2 hours). After each 
faculty small group finished rating its assigned section, she read all results aloud. The faculty then 
engaged in discussion. They did not complete the school profile at this meeting; the Committee decided 
to defer it as well as setting priorities to the next school year since not much time remained in the 
current year. 

Principal and Steering Committee feedback forms from IS4. The principal completed a 
feedback form. In response to question 1 , whether the writing program profile was an accurate picture 
of IS4's program ’s strengths and weaknesses, he answered yes. In response to question 2, if the 
outcome of the study was worth the effort, he also answered affirmatively. In response to question 3, 
how the study had benefited or would benefit the school, he said that setting priorities was scheduled 
to be part of professional development activities for the 1999-2000 school year. Ultimately, he 
anticipated that implementing work regarding their priorities and following up on it would improve 
student vmting. In response to question 4, what were suggestions for improving the School Study 
process, he suggested that all grade levels be included in the interview process for gaining an entire 
school focus. 

Three members of the Steering Committee returned a feedback form. In response to question 
1, what was the respondent 's personal time spent to conduct the study, the chair reported 1 1 hours, 1 
spent 7 hours, and 1 spent 6. In response to question 2, which tasks in the School Study required the 
most time, 1 said reading the interview responses several times ( 1 hour) and summarizing the interview 
responses for the report (3 hours); 1 answered verbatim, but without the time designations; and 1 said 
compiling survey [interview] responses (4 hours). Thus all 3 named the report. 

In response to question 3, how many Steering Committee meetings were held, the Committee 
held 1 planning meeting, at which the chair explained the study’s purpose, shared the completed 
interviews and other materials, and assigned report sections. One respondent also listed an informal 
meeting. 
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In response to question 4, at what points in the study did they need to contact someone with 
questions, 1 said while compiling the report, but the respondent did not indicate whom she or he had 
asked. To the second part of question 4, for which tasks did they need more assistance than was 
available, 2 said none and the third response was ambiguous in meaning and not able to be classified. 

In answer to question 5, whether the school profile was an accurate picture of their writing 
program ’s strengths and weaknesses, all 3 respondents said yes. In answer to question 6, if the outcome 
of the study was worth the effort, 2 said yes and 1 was not sure yet. 

In answer to question 7, how the study had affected or would affect the school, all 3 respondents 
referred to increased participation in writing instruction by nonaccountable teachers. One elaborated, 
“I think . . .[nonaccountable] teachers realized the discrepancies between what they did in regard to 
writing and what the . . . [accountable] grade had to do to produce writing portfolios for our state 
assessment.” Another answered, “Some teachers will begin to use their writing handbook even if 
they’re not ... [an accountable] grade teacher.” 

In answer to question 8, which parts of the study process were most helpful, 2 said the 
interviews. One said faculty discussion, i.e., “Large group of entire staff discussing results was an eye- 
opener for some of the staff Refocus.” 

In answer to question 9, which parts of the study process were least helpful, 1 said none; 1 
offered none, saying all parts were helpful; and 1 said the lack/quality of input from parents or the 
community. 

Question 10 solicited suggestions for improvements to the study process. Two suggested 
including a greater number and/or greater variety of interviews from all grades and content areas. One 
left it blank. 

Question 1 1 asked about the utility of the sample report as an aid to report writing. All 3 
respondents said it was helpful and recommended continuing to include it in the School Study 
handbook. 

Reliability of IS4’s report. IS4 submitted 7 teacher interviews and 12 student interviews, 
neither set of which was compiled, as well as the principal and district administrator interviews. 

One report topic response was entirely equivalent to the reviewer’s sample in accuracy and 
breadth of data items considered. One response was partially equivalent, even though it excluded three 
major points and the detail it included was not the most illustrative of the data being considered. Three 
of the 5 report responses were barely or not at all equivalent to the reviewer’s sample in accuracy and 
scope. They excluded several major and minor points and were unusually brief incorporating either 
no or irrelevant detail and abbreviating statements under a few of the role group spaces to one sentence. 
One of these 3 responses contained writing in the “Other” space, but in none of the role group spaces. 
In the rest of the report, the “Other” space was used in one other instance. Despite the presence of 
several omission errors as described, there were no misrepresentations of data in any of the 5 responses. 
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Overall, the report responses were very brief and contained insufficient detail for capturing and 
characterizing aspects of the writing program. 

Independent School #5 

Description of ISS's study process. IS5 completed its school study in 6 weeks between April 
19 and June 3, 1999. Three months of calling between the contact person and the school, beginning on 
January 22, prefaced the start of the study. 

The contact person said that the school needed substantial help in the beginning. She believed 
that the staff were merely unsure of the process, not realizing how easy it was. Early in February before 
the study began, the principal called the contact person to request that she facilitate the study rather than 
simply advise them as a contact person, to which she agreed. The Steering Committee chair had many 
questions and required assistance getting started and developing the calendar for completion. 
According to the contact/facilitator, the chair is a busy teacher who helps other faculty. Although busy, 
she was motivated to participate in the study because she wanted help with the school’s writing 
program. The contact/facilitator’s first meeting with the school on April 19 was with the principal. 
Steering Committee chair, and curriculum supervisor only, not with the entire Steering Committee. 

The Steering Committee first met on May 3 to preview the study process.'’ They concentrated 
on the “Time and Task Summary” from the handbook to help them frame the process. There were 6 
on the committee, including the chair (a language arts teacher), another language arts teacher, 3 content 
area teachers outside language arts, and 1 parent from the SBDM Council. At some point, one 
individual left the Committee. The parent was scheduled to become part of the Committee activities 
only when it was time to write the report; the chair reviewed the handbook with the parent by phone 
since that individual was not present at the first meeting. Only one other meeting was held, in which 
3 members selected the interviewees. For these meetings and later activities in the study, the 
Committee met during lunch and after school. 

The Steering Committee asked the contact/facilitator to serve as the interviewer and she 
assented. It was a greater level of involvement than the contact/facilitator had anticipated or wanted, 
and she felt that the interviews would have been completed even had she not been the interviewer. 
According to the contact/facilitator, ISS's reason for asking her was that they did not want an outsider; 
they wanted someone whom they could trust (the contact/facilitator was known to the school from 
previous professional interaction). Also, they chose the contact/facilitator because she understood the 
writing program jargon of the interview questions. A parent (not the parent on the Committee) was 
asked to accompany the contact/facilitator to scribe the interviews. The Committee believed that had 
the parent been the interviewer, she might have had difficulty with the specialized jargon used in the 
questions and responses of student interviewees; hence, the Committee asked her to be the scribe 
instead. 



”lf the study’s start is dated from here, then it took just 4 weeks for the school to complete the study; however, the 
Steering Committee chair had done some prior initial work, so the study is dated from April 19 when she and administrators 
first met with the contact person. Also, the chair herself defined the time frame as 6 weeks. 
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IS5 hoped to include a few grades of its feeder school in the study process. The principal and 
faculty of the feeder school were initially reluctant. When IS5 apprised the superintendent of the 
situation, the superintendent explained to the feeder school that the study process was meant to help the 
schools rather than judge them. The feeder school then agreed to participate. The contact/ facilitator 
reported that once interviews were underway at the feeder school, teachers there were helpful and 
positive about the process. The Steering Committee chair indicated that the interviewer eventually 
spoke to teachers of only one of the grades at the feeder school. 

According to the Steering Committee chair, the interviewer and scribe spent 1 day at each of 
the two schools within a 1 -week period. They interviewed a total of4 teachers and 21 students. AtIS5, 
they were able to complete two teacher interviews within 1 hour. They solicited parental permission 
to interview students. Alternates were interviewed when a county chorus event precluded the 
participation of some ofthe original selectees. The student sample included students of varying abilities 
as intended, according to the Committee chair. Student interviews took less time than expected. The 
interviewer and scribe began interviewing students on an individual basis, but when they became 
pressed for time, they switched to each conducting an individual interview simultaneously in the same 
room. The parent was concerned that students would not respond openly in pair interviews. However, 
they did eventually conduct one group student interview and reported that it yielded interesting dialogue 
with no single student dominating the interview. An instructional assistant spent VA days typing and 
compiling student interviews onto 3 separate forms according to grade level; teacher interviews were 
not typed due to time constraints, but the principal interview was. 

IS5 was still in the process of completing the interviews when the first phone interview with the 
chair took place in late May. According to the chair, the contact had encouraged the Committee to 
finish step 3 and some of step 4 before the close of the year, but timing was so poor with state testing 
and portfolio scoring taking place that completion was not feasible. The chair suggested that the fall 
would have been better for conducting all steps in the study. Yet IS5 did make progress into the first 
part of step 4, setting priorities, during the summer. 

To gather the details of the remainder of the study, a second interview with the chair transpired 
in July. The Steering Committee had planned to write the report in a group meeting, after testing was 
completed, near the end of the school year. Ten teachers, including Steering Committee members, from 
both IS 5 and the feeder school were invited to the report- writing meeting; however, none showed up 
except the chair. The chair attributed the lack of participation to the timing of the study. Therefore, the 
chair wrote the report herself, spending at least 3 days on it. (However, a feedback form from another 
participant and the cover sheet to the report both indicated that one other individual contributed a few 
hours to the report.) The chair described the process of writing the report. She said that the findings 
were controversial and supported what they already knew. She tried to be careful, not wanting to “point 
fingers.” She found the student and administrator interview responses to be powerful; the administrator 
responses in particular aided her attempt to not cast blame, in that their answers spoke for her in 
addressing controversial subjects. The same individual who typed student interviews also typed the 
report; the time spent is unknown. 
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The faculty rating meeting was scheduled for a date after the close of school, to involve only 
faculty of grades focused on in the study (5-8). According to the chairpersons, due to IS5's inability 
to pay the participants and the study’s timing, only two other teachers attended besides the chair. The 
chair said that the original Steering Committee members were among those who did not attend; 
therefore, it is unclear whether the two who joined the chair at the rating meeting were other original 
members or late recruits to the Committee. The group worked from 5-1 1 p.m. to complete the ratings 
and write a follow-up report. The chair reported that their dialogue was wonderful, each of them seeing 
things with the benefit of the others’ perspectives. By chance, the superintendent encountered their 
meeting, and stayed to talk with them for an hour. The chair stated that the superintendent was 
encouraging and excited about the study, eager to see what results would develop from it in the 
upcoming school year. 

The follow-up report written by the rating group represented preliminary work on setting 
priorities, which the group intended to use as a “springboard” for discussion among teachers of the 
certain grades at the two schools the following year. They hoped to present the findings either on 
opening day or an early release day. They also hoped to be granted regular time for these teachers to 
work together to align the curriculum; they would seek permission for the teachers to use their release 
day, 1 day per month for 3-4 months, for this purpose. 

While the interviews were still in progress and later, after the rating meeting had occurred, the 
chairperson discussed her experience of IS5's relationship with the feeder school and her insights about 
what she thought the study had done to spur improvement to the writing program. She stated that some 
teachers at the feeder school had not been teaching writing at all because their school’s rewards were 
tied only to the performance of the accountable grade at their own school, but not at all to the 
subsequent grade levels at IS5. The chair said that the School Study was a great thing for IS5 because 
it displayed in an organized fashion the problem that they were already aware of — inadequate writing 
instruction at the feeder school to prepare students for IS5. IS5 had broached the issue with the feeder 
school in the past, yet the feeder school had been reluctant to change the parameters of its writing 
program. Now, since the data bore out IS5's longstanding beliefs about changes that were needed, the 
chair felt that she could “build the bridge” with the feeder school because she could refer the faculty 
to the study’s findings rather than criticize them. Based on IS5's experience, the chair feels strongly 
that the accountability system is flawed because it ties rewards to school buildings instead of grade level 
sections, hence the previous unwillingness of the feeder school to participate more in-depth in the 
writing program. 

Although priorities were not slated to be finalized until the fall, IS 5 acted on the results of the 
writing study during the summer. They obtained the idea from the blood drive example in a collection 
of stories in the School Study handbook. Based on this idea, IS5 students wrote a community resource 
book, a newspaper editorial and advertisements, and radio spots to gamer support for the local Habitat 
for Humanity chapter. The Steering Committee chairperson reported that the radio studio welcomed 
the students; the students read their own ads, which were aired. One of the students lived in a Habitat 
home; he became a “resident expert” on Habitat procedures, a source of pride for him. The chair said 
that the students loved this experience. 
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Half of the $ 1 ,500 stipend was allotted to the chair and the other half equally divided among 2 
Steering Committee members and the person who did the word processing. The parent 
interviewer/Committee member participated as a volunteer and did not share in the stipend. It is 
unknown if the stipend was divided a priori or after the school study was completed based on each 
participant’s contribution; if a priori, it is possible that the difference in allotments had an effect on the 
levels of (non)participation by various individuals. 

Principal and Steering Committee feedback forms from IS5. The principal completed a 
feedback form. In response to question 1 , whether the school profile was an accurate refiection of IS5 's 
writing program ’s strengths and weaknesses, he said yes. In response to question 2, if the outcome of 
the study was worth the effort, he answered the same. In response to question 3, how the study had 
benefited or would benefit the school, he said that it helped them identify their strengths and weaknesses 
and concentrate their efforts on those areas. In response to question 4, what were suggestions for 
improving the School Study process, he had none, saying that the Steering Committee chair seemed 
pleased with it. 

The chair and the 2 other Steering Committee members who participated in the rating meeting 
returned a feedback form, the Committee having dwindled from 6 originally. Responses to several of 
the questions were identical. In response to question 1 , what was the respondent ’s personal time spent 
to conduct the study, the chair said 34 hours, 1 said 6 hours, and 1 said 4 hours. In response to question 
2, which tasks in the School Study required the most time, all 3 respondents said the rating meeting (4 
hours) and 2 mentioned the report as well (10 hours and 2 hours, respectively). One respondent, the 
chair, also listed coordinating interviews and meeting with feeder school faculty (4 hours). 

In response to question 3, how many Steering Committee meetings were held, the Committee 
held 1 organizational meeting prior to the rating meeting. The respondents also listed the rating meeting . 
and a subsequent meeting at which the school profile was completed and findings discussed. 

In response to question 4, at what points in the study did they need to contact someone with 
questions, the chair answered (a) when she needed to find out if it was permissible to change the 
Steering Committee after a member dropped out, and (b) when she needed to know if it was required 
to fill out a consolidated planning form. The other 2 respondents indicated that they had not needed to, 
1 adding that all questions were answered by the chair. To the second part of question A, for which 
tasks did they need more assistance than was available, no respondents answered. 

In answer to question 5, whether the school profile was an accurate picture of their writing 
program ’s strengths and weaknesses, all 3 respondents said yes. In answer to question 6, if the outcome 
of the study was worth the effort, all 3 said yes, 2 adding that it gave supporting evidence for 
weaknesses that they already knew existed. 

In answer to question 7, how the study had affected or would affect the school, all 3 respondents 
said they hoped to secure administrative support and/or funding to correct their weaknesses. One 
respondent, the chair, listed three specific changes that they hoped for: (a) hiring of a writing resource 
teacher; (b) funding for professional development in which teachers from the feeder school and IS5 
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could align curriculum; and (c) an upgrade to permanent status for IS5's English teaching assistant 
position. 

In answer to question 8, which parts of the study process were most helpful, all 3 respondents 
said (a) student interviews, 1 adding because student comments were “honest and revealing,” and (b) 
the dialogue between them during the rating meeting. One respondent, the chair, also listed 
communication with the principal. 

In answer to question 9, which parts of the study process were least helpful, all 3 respondents 
left the item blank. 

Question 10 solicited suggestions for improvements to the study process. All 3 respondents 
suggested conducting the study earlier in the school year, 1 suggesting that the ideal time was the fall. 
Also, all 3 respondents recommended reorganizing interview questions to more closely match the report 
form or report questions to more closely match the rating form. 

Question 1 1 asked about the utility of the sample report as an aid to report writing. All 3 
indicated that they had used it, saying it was helpful. Also, all 3 respondents recommended that the 
sample have more complete responses to report topics. 

Reliability of ISS's report. IS5 submitted 4 teacher interviews, 2 each from IS5 and IS5's 
feeder school, and two compilations of 2 1 student interviews, one of 1 0 students from the feeder school 
and one of 1 1 students from IS5. 

Three of the 5 report responses were entirely or approximately equivalent to the reviewer’s 
sample in accuracy and breadth of data items considered, containing great to good detail adequate for 
summarizing and characterizing aspects of the writing program. These included most major and minor 
points, although 1 response contained a misrepresentation. The other two report responses were 
partially equivalent, 1 included some supportive detail but captured no major or minor points for one 
of the role groups, and 1 captured almost all points but lacked any helpful detail. Although not for these 
5 responses, the “Other” space was used in several places elsewhere in the report for recording 
additional information. In sum, ISS's report was mostly comprehensive and descriptive. The contact 
person/facilitator’s impression of ISS's report was that it reflected the true situation of the school’s 
writing program. 

Independent School #6 

Description of IS6's study process. IS6 conducted its school study in 9 weeks between March 
2 and May 4, 1999. The contact person called the school approximately four times over the course of 
the study. The Steering Committee chair had no questions during the interviews step, finding the 
handbook guidelines easy to follow. She had a question regarding the report, which she directed to 
someone else on the research team. The chair also requested the availability of someone to answer 
questions on the day of the school’s rating meeting. 
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The Steering Committee chairperson was the assistant principal of curriculum and a writing 
committee advisory member. There were 5 on the Committee — the chair, 2 teachers, 1 parent, and 1 
former student who was studying education. The chairperson had anticipated that the Steering 
Committee would be larger, but she surmised that other faculty were uninterested in participating. The 
Committee held 2 planning meetings. The chair made a copy of the School Study handbook prior to 
the first meeting; the group looked through the handbook and “broke it down” conceptually, assigned 
duties, and set deadlines. The chair commented in a phone interview that the handbook had appeared 
overwhelming when they first saw it but was not confusing once they began using it; however, she 
added that the instructions were not very detailed except for step 1 . 

The Steering Committee members served as the interviewers; the chairperson intentionally 
assigned them interviewees outside of their grade level and content area “comfort zone” and set 1 month 
as the time frame for completion. Twelve teachers were interviewed; the sample included teachers from 
all departments with varying levels of experience and attitudes toward writing. Teacher interviews 
lasted 1 hour each. The number of student interviews is unknown. An error occurred in the student 
interviewing process: one of the interviewers distributed the interview form to 3 of the student 
interviewees to complete as a written survey. These surveys contained many blanks and incorrect 
responses. The chair was displeased since this was not their intended method. Of the other student 
interviewees, the chair was unsure whether they were interviewed individually or in groups. The chair 
was generally disappointed with these interview results as well, inferring that they may have been 
rushed. The interviews were not typed. Some of the interviews were not performed because one of the 
Steering Committee members was diverted by a family crisis. According to the chair, interviewing was 
the most time-consuming task for the Committee. 

Committee members worked separately on their report sections within a 1-week time frame. 
The chairperson stated that the task was “not horrible” once they figured out the approach. An 
administrative assistant typed the report in approximately 10 hours. 

The faculty-wide rating meeting was originally scheduled for April 20, but was postponed due 
to the aforementioned personal crisis. The Steering Committee had hoped to have all faculty at the 
meeting; but even by offering professional development hours and a stipend as incentives, it was 
difficult to persuade them. Only about 1/3 of the faculty attended; moreover, they were not an evenly 
distributed cross-section of grade levels and content areas. The faculty were divided into small groups 
when they arrived at the meeting. The rating meeting lasted about 2 hours — quicker than the 
Committee expected. At first, faculty were slightly confused about the task but understood it once they 
completed one or two ratings. The chair also reported that they had some trouble with the indicators 
not m.atching up between the report and rating form. The session concluded with the rating activity, 
not progressing to extended discussion about priorities. 

The chair mused that it was interesting to see faculty’s reactions to some of the data in the 
report. Namely, some findings illustrated administrators’ unfamiliarity with the writing program. Also, 
the chair added that the principal was surprised about the differences between teachers’ and students’ 
perspectives. 
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The activities of step 4 were assigned to two committees. First they would fall under the 
jurisdiction of the Writing Committee, which is made up of department heads who plan writing. Then 
that Committee’s recommendations would be forwarded to the Consolidated Planning Committee, 
representing a cross-section of departments, to coniplete the writing component of the school plan. 

Principal and Steering Committee feedback forms from IS6. The principal completed a 
feedback form. In response to question 1 , whether the writing program profile was an accurate picture 
of IS6's program 's strengths and weaknesses, he stated that the school profile was somewhat accurate, 
suggesting that perhaps more interviews would have helped in this regard. In response to question 2, 
if the outcome of the study was worth the effort, he said yes. In response to question 3, how the study 
had benefited or would benefit the school, he expected that it would help them improve writing across 
the curriculum. In response to question 4, what were suggestions for improving the School Study 
process, he had no further suggestions for improving the study. 

Four of the 5 participants on the Steering Committee returned a feedback form. In response to 
question 1, what was the respondent ’s personal time spent to conduct the study, 1 spent 40-50 hours, 
1 spent 22 hours, 1 spent 15-20 hours, and 1 spent 6-8 hours. In response to question 2, which tasks 
in the School Study required the most time, 3 of the 4 said the interviews, and 1 said preparing and 
typing the report. Also mentioned were planning and the faculty rating meeting. 

In response to question 3, how many Steering Committee meetings were held, the Committee 
held 2 planning meetings: the first for orientation and assigning responsibilities, and the second for 
reviewing the interviews. 

In response to question 4, at what points in the study did they need to contact someone with 
questions, 2 said at the report-writing step, 1 said at the administrator interviews, and 1 did not respond. 
To the second part of question A, for which tasks did they need more assistance than was available, 1 
responded by saying that the assistance provided (for report writing) was adequate. 

In answer to question 5, whether the school profile was an accurate picture of their writing 
program 's strengths and weaknesses, 2 said yes, 1 said “for the most part,” and 1 said no. To the 
second part of question 5, what inaccuracies were there and what about the study process may have 
caused them, 2 respondents identified the blanks in the student interviews that had been completed as 
surveys rather than verbal interviews. 

In answer to question 6, if the outcome of the study was worth the effort, all 4 responded yes, 
1 adding that it was interesting from her perspective as an English teacher and 1 saying that it 
heightened faculty’s awareness of the program’s strengths and motivation to improve its weaknesses. 

In answer to question 7, how the study had affected or would affect the school, 2 mentioned 
heightened faculty awareness and the study’s use as a starting point for improvement; 1 respondent 
identified the study’s benefit for curriculum alignment; and 1 hoped the study would influence IS6's 
focus on writing and the consolidated plan. 




1-38 

i?5 



In answer to question 8, which parts of the study process were most helpful, 1 said 
understanding others’ perceptions of the writing program; 1 said student input and understanding 
students’ perceptions so that she could improve her communication; 1 said both the interviews and 
report, and the varying, conflicting responses contained in them; and 1 said all parts of the study 
process. 

In answer to question 9, which parts of the study process were least helpful, all 4 respondents 
indicated none. 

Question 10 solicited suggestions for improvements to the study process. One recommended 
a year-end review to assess the study’s impact on student learning. One respondent suggested 
correcting the alignment of indicators between the rating guide and rating form. Two respondents had 
no suggestions. 

Question 1 1 asked about the utility of the sample report as an aid to report writing. All 4 
respondents indicated that the sample report was helpful and recommended continuing to include it in 
the School Study handbook. 

Reliability of IS6’s report. IS6 supplied its report but not its interviews to the collaborative 
research team; therefore, the report could not be examined for reliability. However, an examination of 
the report showed it to be rather comprehensive, seeming to include major points and detail. Also, the 
contact person reported that the school’s report verified what she knew about the school. 
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APPENDIX J: 



'els of All 
: Schools 



Table of Report Sections and Data 



BEST COPY AVAILABLE 
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Kejn 

1. D - District administrator interview 

2. P - Principal interview 

3. T - Teacher composite interview 

4. S - Student composite interview 



School Study of Writing Instruction 
Table of report sections and data reference items (p. 1 of 4) 



: . Report section 


Data reference items 


1 . District Administrative Support 
(report topic lA) 


lA 


D: 1,3-5,7-16 
P: 4,6-8,10,13-16,20,23 
T: 4-5, 10, 12-19 


School Administrative Support 
(report topic IB) 


IB 


P: 1-2,4-9,11-12,14-33 
T; 4-6, 9-11, 13-28, 34, 45, 49-50 


2. Professional Development 
(report topics 2A-E) 


2A 


T: 15-20 
P: 13-21 
D: 1,3-4,9-12 




2B 


same as 2A 




2C 


same as 2A 




2D 


same as 2A 




2E 


T: 15,17,19 
P: 14-17 
D: 9 


Writing Program Coordination 
(report topic 3) 


3 


D: 7-8 
P: 8-12,27 

T: 6,9-14,24-25,30,46 
S: 6-8,23-25 


Focus/Intensity of Writing 
(report topic 7) 


7 


T: 1-5,9-10,13-14,32 
S: 4-7,9-10,32 
P: 1-2,8-12 
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School Study of Writing Instruction 



Table of report sections and ( 


ata reference items (p. 2 of 4) 


Report section 


Data reference items 


3. Scoring 


2F 


D: 


13 


(report topic 2F) 




P: 


17-18, 22-26 






T: 


19,21-23 


Collaboration in Professional 


2G 


D: 


12 


Development (report topics 2G-H) 




P: 


13-14,21,27-29 






T: 


15,24-27 




2H 


P: 


14,16,18-19,27-29 






T: 


15, 17, 20, 24-27 


School Climate 


4 


P: 


1-3, 7, 16, 19-21,26-28 


(report topic 4) 




T: 


5-10, 17-18, 20, 22-26 






S: 


1-2, 4-7, 20, 26-27,31-34 


4. Instructional Strategies I 


8 


T: 


45 


(report topics 8; 9A-E) 










9A 


T: 


4, 6-7, 9-11, 13,30-32,41,46 






S: 


6-8, 19-20, 23-24 






P: 


1-2, 5-6, 8-9, 11 






D: 


8 




9B 


T: 


2-7, 9, 13-14, 30-31,35,41-42 






S: 


1-3, 6-8, 14, 19-20, 23-25,31-33 






P: 


1-2, 4-8, 11-12, 26 






D: 


1-5,7 




9C 


T: 


8 






S: 


1,33-34 






P: 


3 






D: 


6 




9D 


S: 


1-2, 4-5, 19-20, 25-27, 33-35 




9E 


T: 


10, 14, 32, 47-48 






S: 


3, 15-16, 23-25,33 






P: 


8 12 
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School Study of Writing Instruction 
Table of report sections and data reference items (p. 3 of 4) 



Report section 


Data reference items 


5. Instructional Strategies II 


lOA 


T: 


29 


(report topics I OA-D; II A-C) 




S: 


1,3,8-13, 17, 25,28,32-33 




lOB 


T: 


47-48 






S: 


15-16 




IOC 


T: 


33-34 






S: 


9-10 




lOD 


T: 


36-37 






S: 


1 1-13,26-28 




llA 


T: 


43 






S: 


21-22 




IIB 


T: 


39-40, 42, 50 






S: 


18-19,21-22,30 






P: 


33 




lie 


T: 


25,33,44 


6. Instructional Strategies III 


11 


T: 


38 


(report topics 12; I3A-B; I4A-B) 




S: 


17-19 




13A 


T: 


41 






S: 


19-20 




13B 


T: 


41 






S: 


19-20,25-27 




14A 


T: 


39-40 






S: 


18-19 




14B 


T: 


29,33,35,37 






S: 


II-I4, 28, 31-33 
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School Study of Writing Instruction 



Table of renort sections and d 


ata reference items tn. 4 of 41 




Report section 


Data reference items 


7. Family & Community Involvement 
(report topics 5, 6) 


5 D: 14 

P: 31-32 
T: 37,49 
S: 26-29 

6 D: 15 
P: 33 
T: 50 

S: 20, 26-27, 30 


8. Overall view of the value for students 
of the writing portfolio program in its 
current form (report topics 15, 16, 17) 


15 T: 1-10,13-14,29-32,36-38,41-43, 

46-48 

S: 1-2,6-8,11-12,14-17,19-22,31- 
34 

16 S: 1-3,15,23-25,29,31-34 

17 D; refer to other report topics 

P: refer to other report topics 
T: refer to other report topics 
S: refer to other report topics 
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APPENDIX K; 



Table of Observed Score Differences, t-Values, p-Values, and Significance Levels of All 
Indicators Between the Continuously Improving and Continuously Declining Schools 

(corrected version to reflect 36 indicators) 
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Table of Observed Score Differences, t-Values, p-Values, and Significance Levels 
of All Indicators Between the Continuously Improving and Continuously Declining Schools 
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Table of Observed Score Differences (continued) 
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Completed Citation Form from The Program Evaluation Standards (1994) 
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Checklist for Applying the Standards 

To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



Descriptor 

U 1 Stakeholder Identification 


The Standard was 
addressed 

X 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


U2 Evaluator Credibility 


X 








U3 Information Scope and Selection 




X 






U4 Values Identification 


X 








U5 Report Clarity 


X 








U6 Report Timeliness and Dissemination 




X 






U7 Evaluation Impact 


X 








FI Practical Procedures 


X 








F2 Political Viability 






X 




F3 Cost Effectiveness 




X 






PI Service Orientation 


X 








P2 Formal Agreements 






X 




P3 Rights of Human Subjects 


X 








P4 Human Interactions 


X 








P5 Complete and Fair Assessment 


X 






- 


P6 Disclosure of Findings 


X 








P7 Conflict of Interest 


X 








P8 Fiscal Responsibility 


X 








A 1 Program Documentation 


X 








A2 Context Analysis 


X 








A3 Described Purposes and Procedures 


X 








A4 Defensible Information Sources 


X 








A5 Valid Information 


X 








A6 Reliable Information 


X 








A7 Systematic Information 


X 








A8 Analysis of Quantitative Information 


X 








A9 Analysis of Qualitative Information 


X 








A 10 Justified Conclusions 


X 








All Impartial Reporting 


X 








A12 Metaevaluation 


X 









The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 

request for evaluation plan/design/proposal 

evaluation plan/design/proposal 

evaluation contract 

- evaluation report 

other: 



Name 



Paige Parrish 






Position or Title Research Assistant 



(signature) 



Agency 



AEL, Inc. 



Address . 



P.O. Box 1348 » Charleston, WV .25325-1348 



Relation to Document Main Author 



Date 



1/16/00 




(e.g., author of document, evaluation team leader, external auditor, internal auditor) 





U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reoroduction Basis 

K 




This document is covered by a signed "Reproduction Release 
(Blanket)" form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a "Specific Document" Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either "Specific Document" or "Blanket"). 
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